XFMS picks the right LLM model for any stated task. You give it a concrete purpose ("fixing bugs in a Python codebase", "summarizing 50-page commercial leases"), and it infers which quality benchmarks matter, weighs every model in its catalog against those dimensions, and returns a ranked shortlist with plain-English rationale per pick.
The catalog updates continuously from 8 independent third-party evaluators — no provider self-reports, no single-source benchmarks.
Glama MCP Gateway
Connect through Glama MCP Gateway for full control over tool access and complete visibility into every call.
Full call logging
Every tool call is logged with complete inputs and outputs, so you can debug issues and audit what your agents are doing.
Tool access control
Enable or disable individual tools per connector, so you decide what your agents can and cannot do.
Managed credentials
Glama handles OAuth flows, token storage, and automatic rotation, so credentials never expire on your clients.
Usage analytics
See which tools your agents call, how often, and when, so you can understand usage patterns and catch anomalies.
Average 4.5/5 across 5 of 5 tools scored.
Each tool has a distinct, well-defined purpose: pick gives a single answer, rank provides a shortlist, discover reveals criteria, benchmark tests engine-chosen candidates, and compare tests user-specified models. There is no overlap.
All five tool names are single verbs (benchmark, compare, discover, pick, rank), following a consistent imperative convention without mixing styles.
With 5 tools, the server is well-scoped for its purpose—LLM selection and comparison. Each tool represents a distinct stage in the decision process, and the count is neither too small nor too large.
The tools cover the main workflows: discovery, selection, ranking, and live testing. However, a tool to list or inspect the underlying model catalog is missing, which agents may need for context.
benchmarkBenchmark the engine's top picks with real test queriesARun a live A/B test against the engine's TOP 3 PICKS for a stated purpose — the engine chooses the candidates from the full catalog. Generates 5 representative test queries (auto-expands to 10 or 15 if results are too close to call), runs them through the picked models in parallel, and returns real cost, latency, and plain-English commentary on who won what. Use AFTER pick or rank when the user wants the engine's own picks stress-tested with live data. DO NOT use this when the user has already named specific candidate models — the engine will ignore the names and test its own picks. Use compare instead in that case. Costs more than rank (15+ live LLM calls).
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing what the model will be used for. The benchmark generates representative test queries from this — so be concrete, not vague. |
| Name | Required | Description |
|---|---|---|
| models | No | Ranked shortlist of models, highest score first. |
| status | No | |
| ab_result | No | |
| catalog_size | No | |
| filtered_out | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description adds significant context beyond annotations: auto-expansion of test queries (5 to 10/15), parallel execution, returns cost/latency/commentary, and cost note (15+ LLM calls). No contradiction with annotations which indicate read-only, idempotent, non-destructive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is well-structured: core action first, then details, then usage guidelines, then cost note. Every sentence adds value. Slightly long but efficient for the information conveyed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (benchmark with auto-expansion, multiple candidates), the description covers usage context, behavior, and alternatives comprehensively. Output schema exists to detail return values, so description need not include that.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (parameter 'purpose' documented in schema). Description adds value by explaining that purpose is used to generate test queries and advising to be concrete. This goes beyond the schema description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Run a live A/B test' on the engine's top 3 picks. It specifies the resource (the engine's own picks) and differentiates from siblings like 'compare' by explicitly stating when to use each.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says 'Use AFTER pick or rank' and 'DO NOT use when the user has already named specific candidate models... Use compare instead.' Also mentions cost relative to 'rank'. Provides clear when-to-use and when-not-to-use with alternative.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
compareCompare specific models head-to-head with real test queriesARun a live A/B test between 2–5 user-specified models for a stated purpose. NO ranking step — the supplied model_ids ARE the candidate set. Generates 5 representative test queries from the purpose, runs them through every named model in parallel, and returns real cost, latency, and plain-English commentary on who won what. Unknown IDs are dropped with a note; if fewer than 2 IDs resolve, the call refuses. Use this whenever the user names specific models to compare (e.g. 'A/B test X and Y'). For engine-chosen candidates, use benchmark instead. Costs more than rank (10+ live LLM calls). Free-tier note: when any candidate ends in ':free', the probe is capped at 3 queries (no adaptive expansion) because free-tier rate limits often push longer probes past the deploy's 5-minute ceiling — evidence will be shallower. The commentary surfaces this when it happens.
| Name | Required | Description | Default |
|---|---|---|---|
| primary | No | Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first. | |
| purpose | Yes | One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague. | |
| model_ids | Yes | Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case. |
| Name | Required | Description |
|---|---|---|
| status | No | |
| purpose | No | |
| ab_result | No | |
| refusal_reason | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
| model_ids_tested | No | |
| invalid_model_ids | No | |
| model_ids_requested | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, destructiveHint=false, which aligns with description. Description adds behavioral details: generates 5 test queries, parallel execution, returns cost/latency/commentary, handling of unknown IDs, refusal condition, and free-tier note about query cap and evidence shallowness.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is concise and well-structured: front-loads main action, covers key behaviors, usage guidelines, and edge cases without unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 3 parameters, 100% schema coverage, annotations, and output schema, the description is fully complete: it explains purpose, usage, behavioral nuances, edge cases, and limitations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and description adds meaning beyond schema: clarifies 'primary' only affects commentary, 'purpose' generates queries, and 'model_ids' details dropping unknown IDs and min/max range with examples.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool compares specific models head-to-head with real test queries, using the verb 'compare' and explicitly distinguishes from sibling tools like 'benchmark' and 'rank'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit guidance on when to use this tool (user names specific models) vs alternatives (use 'benchmark' for engine-chosen candidates), and explains constraints like 2-5 models, unknown ID handling, and cost implications.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
discoverDiscover quality dimensionsAShow which quality dimensions matter for a stated purpose, WITHOUT ranking any models. Returns the inferred weights and the discovery-walk trace. Useful for understanding how XFMS interprets the purpose before committing to a pick.
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing the task. The tool returns which quality dimensions XFMS would weigh for this purpose, without actually ranking any models. Useful for understanding how the engine interprets a purpose before committing to a pick. |
| Name | Required | Description |
|---|---|---|
| events | No | Trace of the discovery walk. |
| weights | No | Per-dimension weights inferred for this purpose. |
| derived_purpose | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, destructiveHint=false, and idempotentHint=true, providing a safe behavioral profile. The description adds value by explaining that the tool does not rank models and returns 'inferred weights and the discovery-walk trace,' which are not covered by annotations. No contradictions, and it adds meaningful context beyond the structured fields.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loads the key action and differentiator, and includes the return value and use case. Every sentence provides essential information without redundancy or filler. Excellent structure for quick comprehension.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (1 parameter, no nested objects, annotations present, output schema exists), the description is complete. It explains the purpose, what it returns, and when to use it. The output schema covers return values, so the description does not need to elaborate further.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with the input schema's description for 'purpose' already explaining the tool's function thoroughly. The overall description adds broader context but does not introduce new parameter-specific semantics beyond what is in the schema. Baseline score of 3 is appropriate as the description adds marginal extra value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The tool name 'discover' and title 'Discover quality dimensions' are clear. The description states a specific action: 'Show which quality dimensions matter for a stated purpose' and explicitly distinguishes from sibling tools like 'rank' by stating 'WITHOUT ranking any models.' This provides a clear understanding of the tool's unique role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description says it's 'useful for understanding how XFMS interprets the purpose before committing to a pick,' implying use before ranking or picking. It distinguishes from siblings without explicit exclusions, but the context makes it clear that if you need actual rankings, you should use another tool. The lack of explicit when-not-to-use prevents a higher score.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
pickPick the best LLMAReturn the single best LLM for a stated purpose. Concise output, no list. Use when the user has settled on the criteria and just wants one answer.
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing what the model will be used for. Be concrete, not vague: 'summarizing 50-page commercial leases' works; 'summarization' does not. |
| Name | Required | Description |
|---|---|---|
| name | No | |
| model_id | No | |
| provider | No | |
| rationale | No | |
| total_score | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false, indicating safe read-only behavior. The description adds behavioral info like 'Concise output, no list' and 'Return the single best LLM', which are consistent with annotations. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words. The first sentence states the purpose, the second provides usage guidance. Front-loaded with the core action and output format.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a single well-described parameter, an output schema, and rich annotations, the description covers all necessary aspects: what the tool does, when to use it, and what to expect (concise output). No gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with the parameter 'purpose' already having a detailed description. The tool description mentions 'for a stated purpose' but does not add significant new meaning beyond the schema. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns 'the single best LLM for a stated purpose', with a specific verb and resource. It also distinguishes from siblings by emphasizing 'Concise output, no list' and 'when the user has settled on the criteria and just wants one answer', contrasting with sibling tools like 'rank' or 'compare'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use when the user has settled on the criteria and just wants one answer', providing clear context for when to use this tool. However, it does not include explicit exclusions or mention of alternative tools, though sibling names are available for the agent to infer.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
rankRank LLMsARank LLMs for a stated purpose. Returns a shortlist with weights, scores, and plain-English rationale per pick. Use when the user wants to see and compare alternatives, not just one answer.
| Name | Required | Description | Default |
|---|---|---|---|
| top_n | No | How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives. | |
| primary | No | Mark dimensions as primary tier. When set, the engine switches from weighted-sum blending to lexicographic ordering: the primary dimension is the sole ranking axis, and other dimensions only break ties. Use when the user says 'cheapest model, period' or similar — their stated preference becomes sacrosanct. | |
| purpose | Yes | One sentence describing what the model will be used for. Be concrete, not vague: 'fixing bugs in a Python codebase' works; 'coding' does not. The more specific the purpose, the better XFMS can infer which quality dimensions matter. | |
| capabilities | No | Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use. |
| Name | Required | Description |
|---|---|---|
| models | No | Ranked shortlist of models, highest score first. |
| status | No | |
| catalog_size | No | |
| filtered_out | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true, and openWorldHint=true, so the description does not need to repeat these. The description adds behavioral context by stating that the tool returns weights, scores, and plain-English rationale, which goes beyond the annotations. No contradictions are present.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, with no extraneous words. The first sentence states the core function, and the second provides usage guidance. Every word earns its place, making it highly concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of the tool (ranking with optional lexicographic ordering), the description is fairly complete. It explains the return format and usage context. An output schema exists (though not shown), and annotations cover safety. Minor gaps include lack of discussion on error handling or edge cases, but overall it is sufficient for the agent to select and invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with detailed descriptions for each parameter. The tool description does not add any parameter-specific meaning beyond what the schema provides, so the baseline of 3 is appropriate. The description does not repeat or expand on parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Rank' and resource 'LLMs', and specifies what is returned (shortlist with weights, scores, rationale). It distinguishes from siblings by noting 'use when the user wants to see and compare alternatives, not just one answer', differentiating from a tool like 'pick' that likely returns a single answer.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes an explicit usage cue: 'Use when the user wants to see and compare alternatives, not just one answer.' This implies when not to use (for single answers) and helps the agent choose between this and sibling tools. However, it does not explicitly name sibling alternatives or provide exclusions, which would elevate it to a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Claim this connector by publishing a /.well-known/glama.json file on your server's domain with the following structure:
{
"$schema": "https://glama.ai/mcp/schemas/connector.json",
"maintainers": [{ "email": "your-email@example.com" }]
}The email address must match the email associated with your Glama account. Once published, Glama will automatically detect and verify the file within a few minutes.
Control your server's listing on Glama, including description and metadata
Access analytics and receive server usage reports
Get monitoring and health status updates for your server
Feature your server to boost visibility and reach more users
For users:
Full audit trail – every tool call is logged with inputs and outputs for compliance and debugging
Granular tool control – enable or disable individual tools per connector to limit what your AI agents can do
Centralized credential management – store and rotate API keys and OAuth tokens in one place
Change alerts – get notified when a connector changes its schema, adds or removes tools, or updates tool definitions, so nothing breaks silently
For server owners:
Proven adoption – public usage metrics on your listing show real-world traction and build trust with prospective users
Tool-level analytics – see which tools are being used most, helping you prioritize development and documentation
Direct user feedback – users can report issues and suggest improvements through the listing, giving you a channel you would not have otherwise
The connector status is unhealthy when Glama is unable to successfully connect to the server. This can happen for several reasons:
The server is experiencing an outage
The URL of the server is wrong
Credentials required to access the server are missing or invalid
If you are the owner of this MCP connector and would like to make modifications to the listing, including providing test credentials for accessing the server, please contact support@glama.ai.
No comments yet. Be the first to start the discussion!
