XFMS — Xpansion Framework Model Source

Compare specific models head-to-head with real test queries

compare

Read-onlyIdempotent

Run a live A/B test between 2–5 user-specified models for a stated purpose. NO ranking step — the supplied model_ids ARE the candidate set. Generates 5 representative test queries from the purpose, runs them through every named model in parallel, and returns real cost, latency, and plain-English commentary on who won what. Unknown IDs are dropped with a note; if fewer than 2 IDs resolve, the call refuses. Use this whenever the user names specific models to compare (e.g. 'A/B test X and Y'). For engine-chosen candidates, use benchmark instead. Costs more than rank (10+ live LLM calls). Free-tier note: when any candidate ends in ':free', the probe is capped at 3 queries (no adaptive expansion) because free-tier rate limits often push longer probes past the deploy's 5-minute ceiling — evidence will be shallower. The commentary surfaces this when it happens.

Input Schema

TableJSON Schema

Name	Required	Description
`primary`	No	Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first.
`purpose`	Yes	One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague.
`model_ids`	Yes	Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case.

Output Schema

TableJSON Schema

Name	Required	Description
`status`	No
`purpose`	No
`ab_result`	No
`refusal_reason`	No
`xpansion_update`	No	Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.
`model_ids_tested`	No
`invalid_model_ids`	No
`model_ids_requested`	No

TDQS

A5/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate readOnlyHint=true, destructiveHint=false, which aligns with description. Description adds behavioral details: generates 5 test queries, parallel execution, returns cost/latency/commentary, handling of unknown IDs, refusal condition, and free-tier note about query cap and evidence shallowness.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is concise and well-structured: front-loads main action, covers key behaviors, usage guidelines, and edge cases without unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 3 parameters, 100% schema coverage, annotations, and output schema, the description is fully complete: it explains purpose, usage, behavioral nuances, edge cases, and limitations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and description adds meaning beyond schema: clarifies 'primary' only affects commentary, 'purpose' generates queries, and 'model_ids' details dropping unknown IDs and min/max range with examples.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool compares specific models head-to-head with real test queries, using the verb 'compare' and explicitly distinguishes from sibling tools like 'benchmark' and 'rank'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit guidance on when to use this tool (user names specific models) vs alternatives (use 'benchmark' for engine-chosen candidates), and explains constraints like 2-5 models, unknown ID handling, and cost implications.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All XFMS — Xpansion Framework Model Source tools Inspect

Try in Browser

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.5/5.0

Disambiguation5/5

Each tool has a distinct, well-defined purpose: pick gives a single answer, rank provides a shortlist, discover reveals criteria, benchmark tests engine-chosen candidates, and compare tests user-specified models. There is no overlap.

Naming Consistency5/5

All five tool names are single verbs (benchmark, compare, discover, pick, rank), following a consistent imperative convention without mixing styles.

Tool Count5/5

With 5 tools, the server is well-scoped for its purpose—LLM selection and comparison. Each tool represents a distinct stage in the decision process, and the count is neither too small nor too large.

Completeness4/5

The tools cover the main workflows: discovery, selection, ranking, and live testing. However, a tool to list or inspect the underlying model catalog is missing, which agents may need for context.

Resources

Need Help?