Pick the right LLM for any task. Ranked shortlist with rationale across 8 evaluators.
Glama MCP Gateway
Connect through Glama MCP Gateway for full control over tool access and complete visibility into every call.
Full call logging
Every tool call is logged with complete inputs and outputs, so you can debug issues and audit what your agents are doing.
Tool access control
Enable or disable individual tools per connector, so you decide what your agents can and cannot do.
Managed credentials
Glama handles OAuth flows, token storage, and automatic rotation, so credentials never expire on your clients.
Usage analytics
See which tools your agents call, how often, and when, so you can understand usage patterns and catch anomalies.
Average 4.5/5 across 5 of 5 tools scored.
Each tool targets a clearly distinct mode of operation: pick returns a single best, rank returns a ranked list, discover explains criteria, compare tests user-specified models, and benchmark tests engine-chosen models. The descriptions explicitly cross-reference each other and give precise usage guidance, eliminating ambiguity.
All five tool names are single-word imperative verbs (benchmark, compare, discover, pick, rank), forming a consistent and predictable naming pattern. No mixing of styles or conventions.
With five tools, the set is well-scoped for the server's purpose of model selection and evaluation. Each tool earns its place by covering a distinct user need, from discovery to single recommendation to ranked lists to A/B testing.
The tool surface covers the full decision flow: understanding what matters (discover), getting one answer (pick), comparing alternatives (rank), and validating with live tests (compare and benchmark). There are no obvious dead ends or missing operations for the stated domain.
benchmarkBenchmark the engine's top picks with real test queriesARun a live A/B test against the engine's TOP 3 PICKS for a stated purpose — the engine chooses the candidates from the full catalog. Generates 5 representative test queries (auto-expands to 10 or 15 if results are too close to call), runs them through the picked models in parallel, and returns real cost, latency, and plain-English commentary on who won what. Use AFTER pick or rank when the user wants the engine's own picks stress-tested with live data. DO NOT use this when the user has already named specific candidate models — the engine will ignore the names and test its own picks. Use compare instead in that case. Costs more than rank (15+ live LLM calls).
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing what the model will be used for. The benchmark generates representative test queries from this — so be concrete, not vague. |
| Name | Required | Description |
|---|---|---|
| models | No | Ranked shortlist of models, highest score first. |
| status | No | |
| ab_result | No | |
| catalog_size | No | |
| filtered_out | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations (readOnlyHint, openWorldHint, idempotentHint), the description reveals behavioral details: auto-expanding test queries from 5 to 10/15, parallel model execution, and the return of cost/latency/commentary. It also warns that the engine will ignore user-specified model names, a non-obvious behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense and front-loaded; the first sentence states the core action, and subsequent sentences provide usage constraints and cost implications. Despite being multi-line, each clause serves a distinct purpose (when, when-not, cost), making it appropriately sized for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers operation, usage context, alternatives, edge cases (auto-expansion), and output highlights (cost, latency, commentary). Given the output schema exists, return-value detail is sufficient, and no critical usage dimension is left unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The sole parameter `purpose` is fully described in the schema (100% coverage), so the description doesn't need to add parameter semantics. It reinforces the purpose-driven query generation ('generates 5 representative test queries from this') but adds no additional format or constraint information beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Run a live A/B test against the engine's TOP 3 PICKS' — a specific verb+resource that immediately declares the tool's function. It explicitly contrasts with siblings by naming `pick`, `rank`, and `compare` for different scenarios, distinguishing it from alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use: 'Use AFTER `pick` or `rank`' for stress-testing the engine's own picks. It also provides a clear exclusion: 'DO NOT use this when the user has already named specific candidate models' and directs to `compare` instead, plus cost guidance ('Costs more than `rank`').
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
compareCompare specific models head-to-head with real test queriesARun a live A/B test between 2–5 user-specified models for a stated purpose. NO ranking step — the supplied model_ids ARE the candidate set. Generates 5 representative test queries from the purpose, runs them through every named model in parallel, and returns real cost, latency, and plain-English commentary on who won what. Unknown IDs are dropped with a note; if fewer than 2 IDs resolve, the call refuses. Use this whenever the user names specific models to compare (e.g. 'A/B test X and Y'). For engine-chosen candidates, use benchmark instead. Costs more than rank (10+ live LLM calls). Free-tier note: when any candidate ends in ':free', the probe is capped at 3 queries (no adaptive expansion) because free-tier rate limits often push longer probes past the deploy's 5-minute ceiling — evidence will be shallower. The commentary surfaces this when it happens.
| Name | Required | Description | Default |
|---|---|---|---|
| primary | No | Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first. | |
| purpose | Yes | One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague. | |
| model_ids | Yes | Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case. |
| Name | Required | Description |
|---|---|---|
| status | No | |
| purpose | No | |
| ab_result | No | |
| refusal_reason | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
| model_ids_tested | No | |
| invalid_model_ids | No | |
| model_ids_requested | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark the tool read-only and idempotent, but the description adds significant behavioral detail: it generates 5 test queries, runs them in parallel, drops unknown IDs with a note, refuses when fewer than 2 resolve, and caps free-tier probes to 3 queries. This goes far beyond what the annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Though longer than minimal descriptions, every sentence adds a distinct fact: core behavior, exclusion, cost, refusal condition, and free-tier caveat. It is front-loaded with the primary function and is well-structured for its complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description doesn't need to spell out return details, but it still covers what's returned (cost, latency, commentary), refusal behavior, and the free-tier cap. It is complete for the tool's complexity and edge cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds meaning by clarifying that purpose is used only for query generation (not catalog ranking) and that model_ids are the exact candidate set with unknown-ID drop behavior. This pushes it to a 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Run a live A/B test between 2–5 user-specified models for a stated purpose,' giving a specific verb, resource, and scope. It explicitly distinguishes itself from siblings by declaring 'NO ranking step' and referencing benchmark for engine-chosen candidates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states exactly when to use: 'Use this whenever the user names specific models to compare' and provides an explicit alternative: 'For engine-chosen candidates, use benchmark instead.' It also notes the cost difference from rank and the refusal behavior when fewer than 2 IDs resolve, offering concrete context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
discoverDiscover quality dimensionsAShow which quality dimensions matter for a stated purpose, WITHOUT ranking any models. Returns the inferred weights and the discovery-walk trace. Useful for understanding how XFMS interprets the purpose before committing to a pick.
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing the task. The tool returns which quality dimensions XFMS would weigh for this purpose, without actually ranking any models. Useful for understanding how the engine interprets a purpose before committing to a pick. |
| Name | Required | Description |
|---|---|---|
| events | No | Trace of the discovery walk. |
| weights | No | Per-dimension weights inferred for this purpose. |
| derived_purpose | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false, so the safety profile is covered. The description adds that it returns 'inferred weights and the discovery-walk trace' and explicitly states it does not rank models, providing extra behavioral context beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, with the main verb and scope front-loaded in the first sentence. Every word contributes: it states the function, the explicit non-ranking constraint, the return value, and the use case, with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a single required parameter, a high-coverage schema, a present output schema, and detailed annotations that cover side effects, the description fully covers the tool's purpose and context. It explains what the tool does, what it returns, and when to use it (before a pick), making it complete for an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the description for the 'purpose' parameter is thorough, explaining what the tool does with the input. The tool description itself does not add much beyond the schema because the schema already contains nearly identical explanatory text, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Show') and defines the tool's resource ('quality dimensions' for a stated purpose). It explicitly contrasts itself with ranking tools ('WITHOUT ranking any models'), clearly distinguishing it from siblings like rank and pick.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description states it is useful before committing to a pick, implying the intended use case. It also clarifies that it does not perform ranking, which differentiates it from alternatives, though it doesn't explicitly name the sibling tools or say when not to use each.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
pickPick the best LLMAReturn the single best LLM for a stated purpose. Concise output, no list. Use when the user has settled on the criteria and just wants one answer.
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing what the model will be used for. Be concrete, not vague: 'summarizing 50-page commercial leases' works; 'summarization' does not. |
| Name | Required | Description |
|---|---|---|
| name | No | |
| model_id | No | |
| provider | No | |
| rationale | No | |
| total_score | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, openWorld, idempotent, and non-destructive behavior. The description adds the behavioral trait of output format ('Concise output, no list'), which is not captured in annotations. This adds useful context without contradicting the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core action, and every sentence earns its place. No redundant wording or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With one simple parameter, rich schema description, output schema present, and comprehensive annotations, the description fully covers the necessary context. It states what the tool does and when to use it, leaving no significant gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a single required parameter 'purpose' that has a detailed description. The main description does not add parameter-specific semantics beyond what the schema already provides, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource: 'Return the single best LLM for a stated purpose.' It clearly distinguishes from siblings like 'rank' by explicitly stating 'no list' and emphasizing a single answer.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit guidance: 'Use when the user has settled on the criteria and just wants one answer.' This clarifies the appropriate context but does not explicitly name alternative tools or when not to use them, though the 'no list' hint inherently differentiates from ranking tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
rankRank LLMsARank LLMs for a stated purpose. Returns a shortlist with weights, scores, and plain-English rationale per pick. Use when the user wants to see and compare alternatives, not just one answer.
| Name | Required | Description | Default |
|---|---|---|---|
| top_n | No | How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives. | |
| primary | No | Mark dimensions as primary tier. When set, the engine switches from weighted-sum blending to lexicographic ordering: the primary dimension is the sole ranking axis, and other dimensions only break ties. Use when the user says 'cheapest model, period' or similar — their stated preference becomes sacrosanct. | |
| purpose | Yes | One sentence describing what the model will be used for. Be concrete, not vague: 'fixing bugs in a Python codebase' works; 'coding' does not. The more specific the purpose, the better XFMS can infer which quality dimensions matter. | |
| capabilities | No | Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use. |
| Name | Required | Description |
|---|---|---|
| models | No | Ranked shortlist of models, highest score first. |
| status | No | |
| catalog_size | No | |
| filtered_out | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate read-only, idempotent, non-destructive behavior. The description adds the return structure (shortlist with weights, scores, rationale) which is not in annotations. It does not contradict annotations and provides useful behavioral context about output, earning a 4 given the lower burden from annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three concise sentences deliver purpose, output, and usage without wasted words. Information is front-loaded with the verb 'Rank' and each sentence serves a distinct function. Ideal size and structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given four parameters, an output schema, and rich annotations, the description covers the core purpose, output format, and when to use it. It could have mentioned the ranking dimensions (cost, quality, etc.) but these appear in the schema's primary parameter, so the description does not need to repeat them. Overall it is complete for the given complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all parameters are fully documented in the schema with detailed descriptions (e.g., top_n defaults, primary behavior, purpose specificity, capabilities). The tool description adds minimal parameter nuance beyond this, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool ranks LLMs for a stated purpose and returns a shortlist with weights, scores, and rationale. This distinguishes it from siblings like 'pick' (single choice) and 'compare' (side-by-side analysis), fulfilling the specificity and differentiation criteria.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage context: 'Use when the user wants to see and compare alternatives, not just one answer.' This also implies when NOT to use (when only one answer is wanted), but it does not explicitly name alternative tools. Since the sibling list is available, this is sufficient but not maximally explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Claim this connector by publishing a /.well-known/glama.json file on your server's domain with the following structure:
{
"$schema": "https://glama.ai/mcp/schemas/connector.json",
"maintainers": [{ "email": "your-email@example.com" }]
}The email address must match the email associated with your Glama account. Once published, Glama will automatically detect and verify the file within a few minutes.
Control your server's listing on Glama, including description and metadata
Access analytics and receive server usage reports
Get monitoring and health status updates for your server
Feature your server to boost visibility and reach more users
For users:
Full audit trail – every tool call is logged with inputs and outputs for compliance and debugging
Granular tool control – enable or disable individual tools per connector to limit what your AI agents can do
Centralized credential management – store and rotate API keys and OAuth tokens in one place
Change alerts – get notified when a connector changes its schema, adds or removes tools, or updates tool definitions, so nothing breaks silently
For server owners:
Proven adoption – public usage metrics on your listing show real-world traction and build trust with prospective users
Tool-level analytics – see which tools are being used most, helping you prioritize development and documentation
Direct user feedback – users can report issues and suggest improvements through the listing, giving you a channel you would not have otherwise
The connector status is unhealthy when Glama is unable to successfully connect to the server. This can happen for several reasons:
The server is experiencing an outage
The URL of the server is wrong
Credentials required to access the server are missing or invalid
If you are the owner of this MCP connector and would like to make modifications to the listing, including providing test credentials for accessing the server, please contact support@glama.ai.
No comments yet. Be the first to start the discussion!
Sign in to create a connector for this server.