XFMS
XFMS helps you choose and live-test AI models for a stated task.
Infer which quality dimensions and weights matter for a described purpose with
discover, without ranking models.Rank LLMs with
rank, returning a shortlist with scores, rationale, capability filters, and optional primary dimensions.Get a single best-model recommendation with
pick, without a list.Stress-test the engine’s top 3 picks with
benchmark, using generated live test queries and returning cost, latency, winners, and commentary.Compare 2–5 user-specified models head-to-head with
compare, including invalid-ID handling and refusal when fewer than 2 models resolve.Use free discovery/ranking; live comparisons use your OpenRouter key and usually cost a few cents.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@XFMS Fixing bugs in our Python codebase."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
XF Model Source (XFMS)
Find the AI model your job really needs, and prove it on your own tasks.
AI is specialized. There are 450+ models on OpenRouter, and the one every vendor calls "the best" was not measured on your job. XF Model Source works inside the assistant you already use (Claude Code, Cursor, Cline, Continue, any MCP client) and does three things:
Finds what your job needs. It asks a few plain questions and turns your answers into the job's real requirements: what goes in, what must come back, the format, the cost ceiling, what must never happen.
Narrows the field. It checks every model against those requirements using independent test scores, each model's published capabilities and price, and the results of our own comparisons, then produces a shortlist.
Proves it on your tasks. A screening round runs about 12 likely models on one of your sample tasks. Any model that breaks a must-have is ruled out, and the best six run all your tasks. You get the winner, the reasons, and every answer so you can check it yourself.
Scores and claims can't tell you whether a model will follow your exact format or avoid inventing a detail. So the test, not the leaderboard, makes the final call, and each shortlisted model shows how it has really done in our own comparisons.
Connect
The server is hosted. There is nothing to install. The address is:
https://xfms.xpansion.dev/mcp
Claude (claude.ai or the desktop app): Settings, Connectors, Add custom connector, paste the address. Sign in with your email when it opens.
Claude Code: run
claude mcp add model-source --transport http https://xfms.xpansion.dev/mcp, then/mcpinside Claude Code to sign in.Codex (in the ChatGPT desktop app): Settings, MCP servers, Add server. Name it
model-source, choose Streamable HTTP, paste the address, save, then select Restart. When it shows Authenticate, select it and sign in with your email.Codex (in a terminal): run
codex mcp add model-source --url https://xfms.xpansion.dev/mcp, thencodex mcp login model-source.Cursor, Windsurf and other assistants: add an HTTP MCP server at the address (Cursor:
~/.cursor/mcp.json):
{
"mcpServers": {
"model-source": { "url": "https://xfms.xpansion.dev/mcp" }
}
}An assistant that can't open a sign-in page: sign in at https://xfms.xpansion.dev/signin and use the personal connection link from your account page as the server address. Treat it like a password.
Sign-in is free and needs only your email. Then ask your assistant: "Help me pick an AI model for pulling the invoice number, date, total and vendor out of invoices as JSON, about 2,000 a month. The cheapest one that gets every field right."
Related MCP server: ai-compass
What it costs
Discovery and the shortlist are free. Sign-in is free and needs only your email.
Comparisons run on your own OpenRouter key, usually a few cents. You see the most a comparison can cost before anything runs, and nothing runs until you say go. Add the key as an
X-OpenRouter-Keyheader. It is used for that run and never stored.XF Model Watch (optional, from $19 a month) re-tests your saved jobs every week and emails you only when a cheaper or better model passes every one of your checks, plus a warning before your model is retired. Solo $19 (3 jobs), Team $49 (10), Agency $149 (40). 14-day money-back. Details: https://xfms.xpansion.dev/watch
Tools
discover (finds the job's real requirements), rank (the full shortlist with the evidence behind it),
pick (the single best proven model), compare (the screening round and test on your tasks), benchmark,
field_results (what has worked for jobs of the same kind), and watch_job (XF Model Watch).
What we keep
No conversations are kept. We log the one-line description you start with and an anonymous ID to count use. Our own daily comparisons are published at https://xfms.xpansion.dev, every answer and every verdict, so the method can be checked.
How the shortlist is built
No vendor claims as proof. A model saying it supports JSON or a long context is a claim. We record how often each model actually kept its claims in our own comparisons.
Missing evidence is shown as missing, never as good.
Your requirements decide. Must-haves filter; priorities order what is left.
The test decides the winner. Methodology in
docs/methodology.md.
Honest limits
A comparison uses a handful of your sample tasks, and a model grades part of the answers. Read a result as strong evidence for your decision, not a guarantee. We can only report on models we have tested, and the ranking before the test is a shortlist, not a verdict.
Part of the Xpansion Framework
XFMS doesn't stand alone — it's the model-selection layer of the Xpansion Framework.
The Xpansion thesis
Humans communicate with intent compressed by contextual experience. AI simply predicts patterns in language. Xpansion is the execution layer that bridges them.
Every sentence a human types carries lifetimes of context that the speaker assumes the other side will decompress — what counts as "good enough," which constraints are non-negotiable, what failures last month taught them, what their house style demands. AI doesn't share that context. It predicts patterns in language, filling in the gaps with whatever's plausible to its training data. The result reads as plausible but isn't intent-honoring: sessions lose context, security holes ship silently, contracts break without warning, and there's no way to verify that what was built actually matches what was asked for. They don't know what they don't know, and neither does AI.
Xpansion closes the gap. It decompresses finite intent upfront, enforces it through code-driven AI behavior, and delivers binary-verified results against the intent across persistent memory that survives every session boundary.
Model Source — the model-selection enforcement
When you say "the best model for this task", you're compressing a lot: what counts as best depends on whether you care about factual reasoning or coherent prose, whether cost matters more than latency, whether you actually need vision or just text, whether the call has to stream, whether a particular benchmark dominates your real workload. AI on its own predicts the pattern — what model do most people pick for queries that look like this? — and gives you a plausible-sounding answer that's often wrong for you.
XFMS does the decompression. It takes your stated purpose, infers which benchmarks actually map to it, honors your stated primary preferences without silently overriding them, surfaces the latent requirements you didn't know to ask about (streaming for real-time chat, vision for OCR), and probes the top picks against your real query to verify the recommendation — not predict it. Then it tells you, in plain English, why it picked what it picked.
One module per enforcement
The rest of the Xpansion stack enforces the same decompress- enforce-verify contract for different parts of the work:
Dispatch (
Dispatch) — runtime task router. Watches what kind of work you're doing and routes it to the right tool.Finite Intent (
XFFI) — turns "build me a feature" into a finite spec with binary terminals before any code gets written. Stops scope drift at the source.Boundary Auditor (
XFBA) — checks every code edit against contracts. Stops broken function signatures and mismatched types from ever reaching production.Systemic Impact Analysis (
XSIA) — maps the blast radius of a proposed change before it lands. Tells you what else might break.Token Conservation (
XFTC) — manages how much of the conversation has to stay in the assistant's working memory. Prevents context loss in long sessions.Execution Audit (
XFXA) — verifies every promise from the spec was actually met before declaring a task done. The final binary check.Memory Tree (
XFMT) — session snapshots that stay searchable across conversations. Your assistant remembers what you decided last week.Security Auditor (
XFSA) — static + AI security scanning on every code edit. Catches secrets, injection paths, and unsafe patterns before they ship.
The full picture, with the rest of the modules, lives at xpansion.dev.
Xpansion is in pre-signup right now. Early access and founding licenses are open at xpansion.dev. XFMS is the first piece to ship public + free — the rest follow.
Local development
git clone https://github.com/VisionAIrySE/XFMS.git
cd XFMS
python3 -m venv .venv
.venv/bin/pip install -e .[dev]
.venv/bin/python -m pytest tests/ -vThe tests mock the HTTP layer so they run offline — no API keys needed to develop.
License
This client library is MIT-licensed. The recommender engine, the
catalog, and the ingestion pipeline are not open source. See
NOTICE for the patent reservation language and the
relationship to the broader Xpansion Framework IP.
Contact
Russ Wright — russ@visionairy.biz
Xpansion Framework — xpansion.dev
Security disclosures — see
SECURITY.md
Available Tools
5 toolsbenchmarkBenchmark the engine's top picks with real test queriesARead-onlyIdempotentInspect
Run a live A/B test against the engine's TOP 3 PICKS for a stated purpose — the engine chooses the candidates from the full catalog. Generates 5 representative test queries (auto-expands to 10 or 15 if results are too close to call), runs them through the picked models in parallel, and returns real cost, latency, and plain-English commentary on who won what. Use AFTER pick or rank when the user wants the engine's own picks stress-tested with live data. DO NOT use this when the user has already named specific candidate models — the engine will ignore the names and test its own picks. Use compare instead in that case. Costs more than rank (15+ live LLM calls).
Inputs: purpose (required). Returns: Rank response with ab_result populated — same shape as `rank` plus live perform…
compareCompare specific models head-to-head with real test queriesARead-onlyIdempotentInspect
Run a live A/B test between 2–5 user-specified models for a stated purpose. NO ranking step — the supplied model_ids ARE the candidate set. Generates 5 representative test queries from the purpose, runs them through every named model in parallel, and returns real cost, latency, and plain-English commentary on who won what. Unknown IDs are dropped with a note; if fewer than 2 IDs resolve, the call refuses. Use this whenever the user names specific models to compare (e.g. 'A/B test X and Y'). For engine-chosen candidates, use benchmark instead. Costs more than rank (10+ live LLM calls). Free-tier note: when any candidate ends in ':free', the probe is capped at 3 queries (no adaptive expansion) because free-tier rate limits often push longer probes past the deploy's 5-minute ceiling — evidence will be shallower. The commentary surfaces this when it happens.
Inputs: purpose (required), model_ids (required), primary. Returns: Result of a head-to-head A/B between user-named models. NOT a rank response — n…
discoverDiscover quality dimensionsARead-onlyIdempotentInspect
Show which quality dimensions matter for a stated purpose, WITHOUT ranking any models. Returns the inferred weights and the discovery-walk trace. Useful for understanding how XFMS interprets the purpose before committing to a pick.
Inputs: purpose (required). Returns: events, weights, derived_purpose, xpansion_update
pickPick the best LLMARead-onlyIdempotentInspect
Return the single best LLM for a stated purpose. Concise output, no list. Use when the user has settled on the criteria and just wants one answer.
Inputs: purpose (required). Returns: The single best model. Same shape as rank's models[0].
rankRank LLMsARead-onlyIdempotentInspect
Rank LLMs for a stated purpose. Returns a shortlist with weights, scores, and plain-English rationale per pick. Use when the user wants to see and compare alternatives, not just one answer.
Inputs: purpose (required), top_n, primary, capabilities. Returns: models, status, catalog_size, filtered_out, +1 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
Each tool has a clearly distinct purpose: discover explains quality dimensions, pick returns a single best, rank returns a shortlist, benchmark tests engine-chosen candidates, and compare tests user-specified candidates. There is no overlap; even the two A/B testing tools are cleanly separated by candidate source (engine vs. user).
All tool names are single-word verbs in lowercase (rank, pick, discover, benchmark, compare), following a consistent imperative style. The pattern is uniform and predictable, with no mixing of conventions.
With 5 tools, the server is well-scoped for its purpose of LLM selection and evaluation. Each tool contributes a distinct step in the workflow without redundancy or bloat.
The tool set covers the full intended workflow: discover criteria, pick or rank, then benchmark engine picks or compare user picks. There are no obvious dead ends or missing operations for the stated purpose.
Maintenance
Related MCP Connectors
Compare LLM API prices, search models and providers, and access reviewed benchmark results.
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
Daily LLM leaderboard from independent benchmarks: ranks, scores, prices, speed and comparisons.
LLM rankings from public benchmarks, model comparisons and source-linked results. Updated daily.
Related MCP Servers
- AlicenseAqualityDmaintenanceProvides access to real-time LLM pricing, speed metrics, and performance benchmarks for over 300 models from Artificial Analysis. It enables users to list, filter, and compare models based on costs, tokens per second, and intelligence indices.239 npm9MIT
- AlicenseDqualityDmaintenanceDescribe your AI use case in plain English, get ranked model recommendations with cost estimates and tradeoff reasoning. Covers 62 models across 29 providers. Available as a web app (BYOK + guest tier) and as an MCP server for Claude Desktop and Cursor — same recommendation engine, two interfaces.11MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to compare LLM benchmarks, get pricing, and receive model recommendations for tasks and budgets.59 PyPI1MIT
- AlicenseAqualityDmaintenanceEnables AI agents to discover, compare, and select the best AI models across multiple providers based on pricing, performance, and capabilities, with real-time cost estimation and benchmarking.91,281 npm1MIT