Skip to main content
Glama

XF Model Source

by dev.xpansion
OverviewDiscussionsAdmin

Server Details

Find the right AI model for your job. Free discovery and ranking; tests on your own key.

If you are the author of this connector, you can claim ownership by verifying the domain or GitHub account it belongs to. Claimed connector authors can inspect health checks, view analytics, and manage their listing.
Status
Healthy
Uptime
98.9% over 60 days
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
Repository
VisionAIrySE/XFMS
GitHub Stars
0
Server Listing
XFMS

TDQS

A4.4/5.0

Scored across 5 tools

Disambiguation5/5

Each tool has a clearly distinct role: discover explains dimensions, pick returns one answer, rank returns a shortlist, compare tests user-specified models, and benchmark tests engine-chosen models. Even the two A/B tools (compare vs. benchmark) are sharply separated by who selects the candidates.

Naming Consistency5/5

All five tools are single imperative verbs (discover, pick, rank, compare, benchmark) with lowercase naming and no mixed conventions. The pattern is simple, predictable, and uniform.

Tool Count5/5

Five tools is well-scoped for a model-source server: enough to cover discovery through selection and validation without bloat. Each tool earns its place and there is no redundant surface.

Completeness5/5

The tool set covers the full workflow: discover to understand what matters, pick for single answers, rank for alternatives, and compare/benchmark for live validation. No obvious lifecycle gaps exist for the domain of selecting and testing LLMs by purpose.

Available Tools

7 tools
benchmarkTest the shortlist on the person's own work
Read-onlyIdempotent
Inspect

The live test: the 5 proven models and 1 promising newcomer from discovery do the person's real tasks side by side, every requirement is judged, and the results pick the winner. First call (session + tasks) returns the estimate and an offer — nothing is spent. Show the person the cost; only when they agree, call again with offer and go=true. Runs on the person's OpenRouter key (X-OpenRouter-Key header). Save the returned report as a local file.

ParametersJSON Schema
NameRequiredDescriptionDefault
goNoThe person approved the estimate. Spends on their key.
offerNoThe offer from the estimate. Pass it with go=true to run.
tasksNoThe person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.
ratingsNoBlind ratings: {task id: {letter: good|ok|poor}}, with test_session.
sessionYesThe sealed session from a finished discover conversation.
overruledNoFailures the person accepts, as 'model id|requirement words'.
test_sessionNoFrom a finished run, to overrule failures without rerunning.
current_modelNoOpenRouter id of the model the job uses now, if any.
runs_per_monthNoRoughly how many times a month the job runs, if known.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

compareTest models the person named on their own work
Read-onlyIdempotent
Inspect

Same live test as benchmark, but on the models the person names (model_ids). Needs a finished discover session so every requirement is judged. Estimate first, then go=true.

ParametersJSON Schema
NameRequiredDescriptionDefault
goNoThe person approved the estimate. Spends on their key.
offerNoThe offer from the estimate. Pass it with go=true to run.
tasksNoThe person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.
ratingsNoBlind ratings: {task id: {letter: good|ok|poor}}, with test_session.
sessionYesThe sealed session from a finished discover conversation.
model_idsNoThe OpenRouter ids of the models the person named.
overruledNoFailures the person accepts, as 'model id|requirement words'.
test_sessionNoFrom a finished run, to overrule failures without rerunning.
current_modelNoOpenRouter id of the model the job uses now, if any.
runs_per_monthNoRoughly how many times a month the job runs, if known.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

discoverWork out what the job really needs
Read-onlyIdempotent
Inspect

Discovery: finds the true requirements of the person's job before any model is chosen. Start with purpose: the reply carries the discovery instructions and the evidence catalogue. YOU ask the person the questions (one thing each, plain words, offering the returned choices); then call again with the sealed session, their answers and your structured updates. Each reply gives progress, rejected updates with reasons, next targets, answer choices from live data and a shortlist. At most 5 rounds. Ends with a confirmation view; call with confirm=true once the person agrees. The person can stop any time (end_now=true). Nothing about the person is stored by XFMS.

ParametersJSON Schema
NameRequiredDescriptionDefault
answersNoThe person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words. For an XFMS question the person already answered earlier, send {"choice": choice id, "from_earlier": their exact earlier words} instead of asking it.
confirmNoThe person agrees with the confirmation view.
contextNoOptional, with the person's consent: short labelled excerpts (sample inputs, a conversation summary). Used for this request only, never kept.
end_nowNoThe person wants to stop and see the best result so far.
purposeNoThe job the person wants an AI model for, in their own words. Needed to start.
sessionNoThe sealed session value from the previous discover response. Pass it back unchanged.
updatesNoWhat you learned from the answers, in the format the first discover reply's instructions give: conclusions, requirements, shapes, open_items, item_moves, relation, usage, survivors_matter, pushback. XFMS checks each against its rules and reports any it rejects.
correctionsNoChanges the person makes in the confirmation view: {requirement_id, field, value}.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

field_resultsWhat has worked for jobs like thisA
Read-onlyIdempotent
Inspect

Results from real comparisons on Model Source for one kind of job (call with no job_type for the list): which models passed, how often, which won, typical cost per task and the most common failure. Counted without anyone's job details. A starting point; test on the person's own work before choosing.

ParametersJSON Schema
NameRequiredDescriptionDefault
job_typeNoOne key from the list returned when called without it.

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, non-destructive behavior, so the bar is lower, and the description still adds meaningful context: results are anonymized aggregates ('Counted without anyone's job details') and are advisory rather than decision-grade. It does not discuss staleness, sample size, or pagination, which would have pushed this higher.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and payload are front-loaded in the opening clause, and the closing caveat is short and earns its place. The first sentence is long and comma-heavy, which slightly hurts scanability but wastes no words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of describing return values and does so explicitly (pass rates, winners, cost per task, common failure), plus it explains the empty-argument case. What is missing is any indication of how much data or how recent the underlying comparisons are.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for the single optional parameter, so the baseline is 3. The description restates the no-argument behavior (get the list of valid job_type values) that the schema already documents, adding no new syntax or format detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific resource (aggregated field comparison results on Model Source) and enumerates the payload (passes, win rates, cost per task, common failure), so an agent knows exactly what it returns. It does not, however, distinguish itself from siblings like benchmark or rank, leaving that differentiation to inference.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives real usage context: call with no job_type to get the available list, treat the output as a starting point, and validate on the person's own work before choosing. That is implied guidance rather than explicit when-to-use/when-not rules, and no sibling alternative is named for cases where this data is insufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

pickPick the best model for a discovered job
Read-onlyIdempotent
Inspect

The single best proven model for a job worked out with discover, with why and the runners-up. Pass the discover session.

ParametersJSON Schema
NameRequiredDescriptionDefault
sessionYesThe sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

rankRank models for a discovered job
Read-onlyIdempotent
Inspect

The full shortlist for a job worked out with discover: 5 proven models and 1 promising newcomer, each with its standing on every requirement, plus near misses and what would change the order. Pass the discover session.

ParametersJSON Schema
NameRequiredDescriptionDefault
sessionYesThe sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

watch_jobWatch this job (XF Model Watch subscribers)AInspect

Saves a finished job for XF Model Watch, which re-tests it as new models arrive and emails the person only when a model passes every must-have and ranks higher on their priorities. Needs a signed-in subscriber. Ask the person first: Model Watch keeps the job and its example tasks (encrypted, deletable any time). Pass consent=true only if they agree. Pass the discover session, and either test_session from a finished comparison or the tasks, plus the model they use now.

ParametersJSON Schema
NameRequiredDescriptionDefault
tasksNoExample tasks, if no test_session.
titleNoA short name for the job, in the person's words.
consentYesThe person agreed to Model Watch keeping this job.
sessionYesThe discover session.
test_sessionNoFrom a finished comparison; carries its tasks.
current_modelNoOpenRouter id of the model the job uses now.
runs_per_monthNoRoughly how many times a month the job runs, if known.

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only tell the agent this is a non-read-only, non-destructive, non-idempotent call. The description goes well beyond that: it discloses the auth requirement (signed-in subscriber), the consent gate, the notification policy (email only on a passing, higher-ranked model), and data handling (job and example tasks kept encrypted, deletable any time). This is exactly the additive context annotations cannot carry, and nothing contradicts them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and consequence are front-loaded in the first sentence, with prerequisites and parameter guidance following in a logical order. The consent and encryption clauses are dense but each carries necessary information, though the final sentence packs several parameter rules into one long construction.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a seven-parameter mutation tool with no output schema, the description covers auth, consent, side effects, retention, and which parameters pair together. It stops short of saying what comes back (a job identifier or confirmation) or what happens on a duplicate watch, which would close the remaining gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline would be 3, but the description adds relational meaning the flat schema cannot: that test_session comes from a finished comparison and carries its tasks, making it an alternative to supplying tasks directly, and that the discover session plus the current model are expected together. The exact syntax of current_model or runs_per_month is left to the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource (saves a finished job for XF Model Watch) plus the distinctive behavior that defines the feature: re-testing as new models arrive and emailing only when a model passes every must-have and outranks on the person's priorities. That is enough to separate it from siblings like pick, rank, and compare, which do not persist a job for ongoing monitoring.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear preconditions (signed-in subscriber, explicitly asking the person first) and a hard rule tying consent=true to their agreement. It also says what to pass along (the discover session, and either test_session or the tasks, plus the current model), but it never names a sibling tool or states when this should be chosen over alternatives such as pick or field_results.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool update
    • Changedwatch_job1 field changed
      • changedInput schema / properties / consent / description
        Previous value: -"The person agreed to Watch keeping this job."New value: +"The person agreed to Model Watch keeping this job."
  2. 2 tool updates
    • Changedbenchmark2 fields changed
      • addedInput schema / properties / current_model
        Added value: +{
+  "description": "OpenRouter id of the model the job uses now, if any.",
+  "type": "string"
+}
      • addedInput schema / properties / runs_per_month
        Added value: +{
+  "description": "Roughly how many times a month the job runs, if known.",
+  "type": "number"
+}
    • Changedcompare2 fields changed
      • addedInput schema / properties / current_model
        Added value: +{
+  "description": "OpenRouter id of the model the job uses now, if any.",
+  "type": "string"
+}
      • addedInput schema / properties / runs_per_month
        Added value: +{
+  "description": "Roughly how many times a month the job runs, if known.",
+  "type": "number"
+}
  3. 1 tool update
    • Addedwatch_job
  4. 2 tool updates
    • Changeddiscover3 fields changed
      • addedInput schema / properties / answers / additionalProperties / anyOf
        Added value: +[
+  {
+    "type": "string"
+  },
+  {
+    "properties": {
+      "choice": {
+        "type": "string"
+      },
+      "from_earlier": {
+        "type": "string"
+      }
+    },
+    "required": [
+      "choice",
+      "from_earlier"
+    ],
+    "type": "object"
+  }
+]
      • removedInput schema / properties / answers / additionalProperties / type
        Removed value: -"string"
      • changedInput schema / properties / answers / description
        Previous value: -"The person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words."New value: +"The person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words. For an XFMS question the person already answered earlier, send {\"choice\": choice id, \"from_earlier\": their exact earlier words} instead of asking it."
    • Addedfield_results
  5. 5 tool updates
    • Changedbenchmark16 fields changed
      • removedInput schema / properties / capabilities
        Removed value: -{
-  "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
-  "items": {
-    "enum": [
-      "vision",
-      "audio_in",
-      "tool_use",
-      "structured_outputs"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}
      • addedInput schema / properties / go
        Added value: +{
+  "description": "The person approved the estimate. Spends on their key.",
+  "type": "boolean"
+}
      • addedInput schema / properties / offer
        Added value: +{
+  "description": "The offer from the estimate. Pass it with go=true to run.",
+  "type": "string"
+}
      • addedInput schema / properties / overruled
        Added value: +{
+  "description": "Failures the person accepts, as 'model id|requirement words'.",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}
      • removedInput schema / properties / primary
        Removed value: -{
-  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
-  "items": {
-    "enum": [
-      "cost",
-      "quality",
-      "latency",
-      "privacy"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}
      • removedInput schema / properties / purpose
        Removed value: -{
-  "description": "One sentence describing what the model will be used for. The benchmark generates representative test queries from this — so be concrete, not vague.",
-  "type": "string"
-}
      • addedInput schema / properties / ratings
        Added value: +{
+  "description": "Blind ratings: {task id: {letter: good|ok|poor}}, with test_session.",
+  "type": "object"
+}
      • addedInput schema / properties / session
        Added value: +{
+  "description": "The sealed session from a finished discover conversation.",
+  "type": "string"
+}
      • addedInput schema / properties / tasks
        Added value: +{
+  "description": "The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.",
+  "items": {
+    "properties": {
+      "covers": {
+        "description": "ids of the requirements this task tests",
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "group": {
+        "description": "optional kind of task, e.g. 'short captions'",
+        "type": "string"
+      },
+      "image_urls": {
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "source": {
+        "description": "real | made up",
+        "type": "string"
+      },
+      "text": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}
      • removedInput schema / properties / test_queries
        Removed value: -{
-  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
-  "items": {
-    "minLength": 1,
-    "type": "string"
-  },
-  "maxItems": 15,
-  "minItems": 1,
-  "type": "array"
-}
      • addedInput schema / properties / test_session
        Added value: +{
+  "description": "From a finished run, to overrule failures without rerunning.",
+  "type": "string"
+}
      • removedInput schema / properties / top_n
        Removed value: -{
-  "default": 5,
-  "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
-  "maximum": 25,
-  "minimum": 1,
-  "type": "integer"
-}
      • changedInput schema / required
        Previous value: -[
-  "purpose"
-]New value: +[
+  "session"
+]
      • addedOutput schema / additionalProperties
        Added value: +true
      • changedOutput schema / description
        Previous value: -"Rank response with ab_result populated — same shape as `rank` plus live performance data from the probe runs."New value: +"Without go: the estimate and offer. With go: the outcome, every model on every requirement, the report."
      • removedOutput schema / properties
        Removed value: -{
-  "ab_result": {
-    "properties": {
-      "aggregates": {
-        "description": "Per-model stats across the runs.",
-        "items": {
-          "properties": {
-            "avg_accuracy": {
-              "type": [
-                "number",
-                "null"
-              ]
-            },
-            "avg_completion_tokens": {
-              "type": "number"
-            },
-            "avg_latency_ms": {
-              "type": "number"
-            },
-            "model_id": {
-              "type": "string"
-            },
-            "model_name": {
-              "type": "string"
-            },
-            "runs": {
-              "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
-              "items": {
-                "properties": {
-                  "error": {
-                    "type": [
-                      "string",
-                      "null"
-                    ]
-                  },
-                  "response_text": {
-                    "type": "string"
-                  },
-                  "test_query": {
-                    "type": "string"
-                  }
-                },
-                "type": "object"
-              },
-              "type": "array"
-            },
-            "success_count": {
-              "type": "integer"
-            },
-            "total_cost_usd": {
-              "type": "number"
-            }
-          },
-          "type": "object"
-        },
-        "type": "array"
-      },
-      "commentary": {
-        "type": "string"
-      },
-      "cost_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "incongruity_detected": {
-        "type": "boolean"
-      },
-      "latency_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "overall_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "queries_executed": {
-        "type": "integer"
-      },
-      "test_queries": {
-        "description": "The generated test prompts that were run.",
-        "items": {
-          "type": "string"
-        },
-        "type": "array"
-      }
-    },
-    "type": "object"
-  },
-  "catalog_size": {
-    "type": "integer"
-  },
-  "filtered_out": {
-    "type": "integer"
-  },
-  "frontier_filtered_out": {
-    "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
-    "type": "integer"
-  },
-  "models": {
-    "description": "Ranked shortlist of models, highest score first.",
-    "items": {
-      "properties": {
-        "model_id": {
-          "type": "string"
-        },
-        "name": {
-          "type": "string"
-        },
-        "provider": {
-          "type": [
-            "string",
-            "null"
-          ]
-        },
-        "rationale": {
-          "type": "string"
-        },
-        "total_score": {
-          "type": "number"
-        }
-      },
-      "type": "object"
-    },
-    "type": "array"
-  },
-  "quality_floor_reason": {
-    "type": [
-      "string",
-      "null"
-    ]
-  },
-  "status": {
-    "description": "'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered).",
-    "type": "string"
-  },
-  "xpansion_update": {
-    "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
-    "properties": {
-      "call_count": {
-        "type": "integer"
-      },
-      "id": {
-        "type": "string"
-      },
-      "message": {
-        "type": "string"
-      },
-      "signup_url": {
-        "type": "string"
-      }
-    },
-    "type": "object"
-  }
-}
    • Changedcompare16 fields changed
      • addedInput schema / properties / go
        Added value: +{
+  "description": "The person approved the estimate. Spends on their key.",
+  "type": "boolean"
+}
      • changedInput schema / properties / model_ids / description
        Previous value: -"Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case."New value: +"The OpenRouter ids of the models the person named."
      • removedInput schema / properties / model_ids / maxItems
        Removed value: -5
      • addedInput schema / properties / offer
        Added value: +{
+  "description": "The offer from the estimate. Pass it with go=true to run.",
+  "type": "string"
+}
      • addedInput schema / properties / overruled
        Added value: +{
+  "description": "Failures the person accepts, as 'model id|requirement words'.",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}
      • removedInput schema / properties / primary
        Removed value: -{
-  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
-  "items": {
-    "enum": [
-      "cost",
-      "quality",
-      "latency",
-      "privacy"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}
      • removedInput schema / properties / purpose
        Removed value: -{
-  "description": "One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague.",
-  "type": "string"
-}
      • addedInput schema / properties / ratings
        Added value: +{
+  "description": "Blind ratings: {task id: {letter: good|ok|poor}}, with test_session.",
+  "type": "object"
+}
      • addedInput schema / properties / session
        Added value: +{
+  "description": "The sealed session from a finished discover conversation.",
+  "type": "string"
+}
      • addedInput schema / properties / tasks
        Added value: +{
+  "description": "The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.",
+  "items": {
+    "properties": {
+      "covers": {
+        "description": "ids of the requirements this task tests",
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "group": {
+        "description": "optional kind of task, e.g. 'short captions'",
+        "type": "string"
+      },
+      "image_urls": {
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "source": {
+        "description": "real | made up",
+        "type": "string"
+      },
+      "text": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}
      • removedInput schema / properties / test_queries
        Removed value: -{
-  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
-  "items": {
-    "minLength": 1,
-    "type": "string"
-  },
-  "maxItems": 15,
-  "minItems": 1,
-  "type": "array"
-}
      • addedInput schema / properties / test_session
        Added value: +{
+  "description": "From a finished run, to overrule failures without rerunning.",
+  "type": "string"
+}
      • changedInput schema / required
        Previous value: -[
-  "purpose",
-  "model_ids"
-]New value: +[
+  "session"
+]
      • addedOutput schema / additionalProperties
        Added value: +true
      • changedOutput schema / description
        Previous value: -"Result of a head-to-head A/B between user-named models. NOT a rank response — no ranking happened, so no scores or rationale. Just probe evidence plus a record of which IDs were resolvable."New value: +"Same as benchmark, for the models the person named."
      • removedOutput schema / properties
        Removed value: -{
-  "ab_result": {
-    "properties": {
-      "aggregates": {
-        "description": "Per-model stats across the runs.",
-        "items": {
-          "properties": {
-            "avg_accuracy": {
-              "type": [
-                "number",
-                "null"
-              ]
-            },
-            "avg_completion_tokens": {
-              "type": "number"
-            },
-            "avg_latency_ms": {
-              "type": "number"
-            },
-            "model_id": {
-              "type": "string"
-            },
-            "model_name": {
-              "type": "string"
-            },
-            "runs": {
-              "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
-              "items": {
-                "properties": {
-                  "error": {
-                    "type": [
-                      "string",
-                      "null"
-                    ]
-                  },
-                  "response_text": {
-                    "type": "string"
-                  },
-                  "test_query": {
-                    "type": "string"
-                  }
-                },
-                "type": "object"
-              },
-              "type": "array"
-            },
-            "success_count": {
-              "type": "integer"
-            },
-            "total_cost_usd": {
-              "type": "number"
-            }
-          },
-          "type": "object"
-        },
-        "type": "array"
-      },
-      "commentary": {
-        "type": "string"
-      },
-      "cost_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "incongruity_detected": {
-        "type": "boolean"
-      },
-      "latency_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "overall_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "queries_executed": {
-        "type": "integer"
-      },
-      "test_queries": {
-        "description": "The generated test prompts that were run.",
-        "items": {
-          "type": "string"
-        },
-        "type": "array"
-      }
-    },
-    "type": "object"
-  },
-  "invalid_model_ids": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "model_ids_requested": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "model_ids_tested": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "purpose": {
-    "type": "string"
-  },
-  "refusal_reason": {
-    "type": [
-      "string",
-      "null"
-    ]
-  },
-  "status": {
-    "enum": [
-      "compared",
-      "refused"
-    ],
-    "type": "string"
-  },
-  "xpansion_update": {
-    "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
-    "properties": {
-      "call_count": {
-        "type": "integer"
-      },
-      "id": {
-        "type": "string"
-      },
-      "message": {
-        "type": "string"
-      },
-      "signup_url": {
-        "type": "string"
-      }
-    },
-    "type": "object"
-  }
-}
    • Changeddiscover12 fields changed
      • addedInput schema / properties / answers
        Added value: +{
+  "additionalProperties": {
+    "type": "string"
+  },
+  "description": "The person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words.",
+  "type": "object"
+}
      • addedInput schema / properties / confirm
        Added value: +{
+  "description": "The person agrees with the confirmation view.",
+  "type": "boolean"
+}
      • addedInput schema / properties / context
        Added value: +{
+  "description": "Optional, with the person's consent: short labelled excerpts (sample inputs, a conversation summary). Used for this request only, never kept.",
+  "items": {
+    "properties": {
+      "label": {
+        "type": "string"
+      },
+      "text": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}
      • addedInput schema / properties / corrections
        Added value: +{
+  "description": "Changes the person makes in the confirmation view: {requirement_id, field, value}.",
+  "items": {
+    "type": "object"
+  },
+  "type": "array"
+}
      • addedInput schema / properties / end_now
        Added value: +{
+  "description": "The person wants to stop and see the best result so far.",
+  "type": "boolean"
+}
      • changedInput schema / properties / purpose / description
        Previous value: -"One sentence describing the task. The tool returns which quality dimensions XFMS would weigh for this purpose, without actually ranking any models. Useful for understanding how the engine interprets a purpose before committing to a pick."New value: +"The job the person wants an AI model for, in their own words. Needed to start."
      • addedInput schema / properties / session
        Added value: +{
+  "description": "The sealed session value from the previous discover response. Pass it back unchanged.",
+  "type": "string"
+}
      • addedInput schema / properties / updates
        Added value: +{
+  "description": "What you learned from the answers, in the format the first discover reply's instructions give: conclusions, requirements, shapes, open_items, item_moves, relation, usage, survivors_matter, pushback. XFMS checks each against its rules and reports any it rejects.",
+  "type": "object"
+}
      • removedInput schema / required
        Removed value: -[
-  "purpose"
-]
      • addedOutput schema / additionalProperties
        Added value: +true
      • addedOutput schema / description
        Added value: +"stage (one | two | confirm | confirmed), questions to relay (each with why), open items, coverage, the confirmation view when ready, and the sealed session."
      • removedOutput schema / properties
        Removed value: -{
-  "derived_purpose": {
-    "type": "string"
-  },
-  "events": {
-    "description": "Trace of the discovery walk.",
-    "type": "array"
-  },
-  "weights": {
-    "description": "Per-dimension weights inferred for this purpose.",
-    "type": "object"
-  },
-  "xpansion_update": {
-    "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
-    "properties": {
-      "call_count": {
-        "type": "integer"
-      },
-      "id": {
-        "type": "string"
-      },
-      "message": {
-        "type": "string"
-      },
-      "signup_url": {
-        "type": "string"
-      }
-    },
-    "type": "object"
-  }
-}
    • Changedpick6 fields changed
      • removedInput schema / properties / purpose
        Removed value: -{
-  "description": "One sentence describing what the model will be used for. Be concrete, not vague: 'summarizing 50-page commercial leases' works; 'summarization' does not.",
-  "type": "string"
-}
      • addedInput schema / properties / session
        Added value: +{
+  "description": "The sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first.",
+  "type": "string"
+}
      • changedInput schema / required
        Previous value: -[
-  "purpose"
-]New value: +[
+  "session"
+]
      • addedOutput schema / additionalProperties
        Added value: +true
      • changedOutput schema / description
        Previous value: -"The single best model — same shape as rank's models[0]. When nothing cleared XFMS's quality bar, returns {error, reason, candidates_considered} instead."New value: +"The top proven pick with why, plus the runners-up."
      • removedOutput schema / properties
        Removed value: -{
-  "model_id": {
-    "type": "string"
-  },
-  "name": {
-    "type": "string"
-  },
-  "provider": {
-    "type": [
-      "string",
-      "null"
-    ]
-  },
-  "rationale": {
-    "type": "string"
-  },
-  "total_score": {
-    "type": "number"
-  },
-  "xpansion_update": {
-    "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
-    "properties": {
-      "call_count": {
-        "type": "integer"
-      },
-      "id": {
-        "type": "string"
-      },
-      "message": {
-        "type": "string"
-      },
-      "signup_url": {
-        "type": "string"
-      }
-    },
-    "type": "object"
-  }
-}
    • Changedrank9 fields changed
      • removedInput schema / properties / capabilities
        Removed value: -{
-  "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
-  "items": {
-    "enum": [
-      "vision",
-      "audio_in",
-      "tool_use",
-      "structured_outputs"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}
      • removedInput schema / properties / primary
        Removed value: -{
-  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
-  "items": {
-    "enum": [
-      "cost",
-      "quality",
-      "latency",
-      "privacy"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}
      • removedInput schema / properties / purpose
        Removed value: -{
-  "description": "One sentence describing what the model will be used for. Be concrete, not vague: 'fixing bugs in a Python codebase' works; 'coding' does not. The more specific the purpose, the better XFMS can infer which quality dimensions matter.",
-  "type": "string"
-}
      • addedInput schema / properties / session
        Added value: +{
+  "description": "The sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first.",
+  "type": "string"
+}
      • removedInput schema / properties / top_n
        Removed value: -{
-  "default": 5,
-  "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
-  "maximum": 25,
-  "minimum": 1,
-  "type": "integer"
-}
      • changedInput schema / required
        Previous value: -[
-  "purpose"
-]New value: +[
+  "session"
+]
      • addedOutput schema / additionalProperties
        Added value: +true
      • addedOutput schema / description
        Added value: +"proven picks (each with its standing on every requirement), one promising newcomer, near misses, what changes if priorities change, and the assumptions."
      • removedOutput schema / properties
        Removed value: -{
-  "catalog_size": {
-    "type": "integer"
-  },
-  "filtered_out": {
-    "type": "integer"
-  },
-  "frontier_filtered_out": {
-    "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
-    "type": "integer"
-  },
-  "models": {
-    "description": "Ranked shortlist of models, highest score first.",
-    "items": {
-      "properties": {
-        "model_id": {
-          "type": "string"
-        },
-        "name": {
-          "type": "string"
-        },
-        "provider": {
-          "type": [
-            "string",
-            "null"
-          ]
-        },
-        "rationale": {
-          "type": "string"
-        },
-        "total_score": {
-          "type": "number"
-        }
-      },
-      "type": "object"
-    },
-    "type": "array"
-  },
-  "quality_floor_reason": {
-    "type": [
-      "string",
-      "null"
-    ]
-  },
-  "status": {
-    "description": "'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered).",
-    "type": "string"
-  },
-  "xpansion_update": {
-    "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
-    "properties": {
-      "call_count": {
-        "type": "integer"
-      },
-      "id": {
-        "type": "string"
-      },
-      "message": {
-        "type": "string"
-      },
-      "signup_url": {
-        "type": "string"
-      }
-    },
-    "type": "object"
-  }
-}
  6. 3 tool updates
    • Changedbenchmark5 fields changed
      • addedInput schema / properties / capabilities
        Added value: +{
+  "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
+  "items": {
+    "enum": [
+      "vision",
+      "audio_in",
+      "tool_use",
+      "structured_outputs"
+    ],
+    "type": "string"
+  },
+  "type": "array"
+}
      • addedInput schema / properties / primary
        Added value: +{
+  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
+  "items": {
+    "enum": [
+      "cost",
+      "quality",
+      "latency",
+      "privacy"
+    ],
+    "type": "string"
+  },
+  "type": "array"
+}
      • addedInput schema / properties / test_queries
        Added value: +{
+  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+  "items": {
+    "minLength": 1,
+    "type": "string"
+  },
+  "maxItems": 15,
+  "minItems": 1,
+  "type": "array"
+}
      • addedInput schema / properties / top_n
        Added value: +{
+  "default": 5,
+  "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
+  "maximum": 25,
+  "minimum": 1,
+  "type": "integer"
+}
      • changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / type
        Previous value: -"number"New value: +[
+  "number",
+  "null"
+]
    • Changedcompare3 fields changed
      • changedInput schema / properties / primary / description
        Previous value: -"Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."
      • addedInput schema / properties / test_queries
        Added value: +{
+  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+  "items": {
+    "minLength": 1,
+    "type": "string"
+  },
+  "maxItems": 15,
+  "minItems": 1,
+  "type": "array"
+}
      • changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / type
        Previous value: -"number"New value: +[
+  "number",
+  "null"
+]
    • Changedrank1 field changed
      • changedInput schema / properties / primary / description
        Previous value: -"Mark dimensions as primary tier. When set, the engine switches from weighted-sum blending to lexicographic ordering: the primary dimension is the sole ranking axis, and other dimensions only break ties. Use when the user says 'cheapest model, period' or similar — their stated preference becomes sacrosanct."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."
  7. 4 tool updates
    • Changedbenchmark5 fields changed
      • addedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy
        Added value: +{
+  "type": "number"
+}
      • addedOutput schema / properties / ab_result / properties / aggregates / items / properties / runs
        Added value: +{
+  "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+  "items": {
+    "properties": {
+      "error": {
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "response_text": {
+        "type": "string"
+      },
+      "test_query": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}
      • addedOutput schema / properties / frontier_filtered_out
        Added value: +{
+  "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+  "type": "integer"
+}
      • addedOutput schema / properties / quality_floor_reason
        Added value: +{
+  "type": [
+    "string",
+    "null"
+  ]
+}
      • addedOutput schema / properties / status / description
        Added value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."
    • Changedcompare2 fields changed
      • addedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy
        Added value: +{
+  "type": "number"
+}
      • addedOutput schema / properties / ab_result / properties / aggregates / items / properties / runs
        Added value: +{
+  "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+  "items": {
+    "properties": {
+      "error": {
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "response_text": {
+        "type": "string"
+      },
+      "test_query": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}
    • Changedpick1 field changed
      • changedOutput schema / description
        Previous value: -"The single best model. Same shape as rank's models[0]."New value: +"The single best model — same shape as rank's models[0]. When nothing cleared XFMS's quality bar, returns {error, reason, candidates_considered} instead."
    • Changedrank3 fields changed
      • addedOutput schema / properties / frontier_filtered_out
        Added value: +{
+  "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+  "type": "integer"
+}
      • addedOutput schema / properties / quality_floor_reason
        Added value: +{
+  "type": [
+    "string",
+    "null"
+  ]
+}
      • addedOutput schema / properties / status / description
        Added value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."
  8. 3 tool updates
    • Changedbenchmark1 field changed
      • changedOutput schema / properties / models / items / properties / provider / type
        Previous value: -"string"New value: +[
+  "string",
+  "null"
+]
    • Changedpick1 field changed
      • changedOutput schema / properties / provider / type
        Previous value: -"string"New value: +[
+  "string",
+  "null"
+]
    • Changedrank1 field changed
      • changedOutput schema / properties / models / items / properties / provider / type
        Previous value: -"string"New value: +[
+  "string",
+  "null"
+]
  9. 1 tool update
    • Addedcompare
  10. 4 tool updates
    • Changedbenchmark1 field changed
      • addedOutput schema / properties / xpansion_update
        Added value: +{
+  "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+  "properties": {
+    "call_count": {
+      "type": "integer"
+    },
+    "id": {
+      "type": "string"
+    },
+    "message": {
+      "type": "string"
+    },
+    "signup_url": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}
    • Changeddiscover1 field changed
      • addedOutput schema / properties / xpansion_update
        Added value: +{
+  "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+  "properties": {
+    "call_count": {
+      "type": "integer"
+    },
+    "id": {
+      "type": "string"
+    },
+    "message": {
+      "type": "string"
+    },
+    "signup_url": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}
    • Changedpick1 field changed
      • addedOutput schema / properties / xpansion_update
        Added value: +{
+  "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+  "properties": {
+    "call_count": {
+      "type": "integer"
+    },
+    "id": {
+      "type": "string"
+    },
+    "message": {
+      "type": "string"
+    },
+    "signup_url": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}
    • Changedrank1 field changed
      • addedOutput schema / properties / xpansion_update
        Added value: +{
+  "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+  "properties": {
+    "call_count": {
+      "type": "integer"
+    },
+    "id": {
+      "type": "string"
+    },
+    "message": {
+      "type": "string"
+    },
+    "signup_url": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}
  11. 1 tool update
    • Addedbenchmark
  12. 3 tool updates
    • First observeddiscover
    • First observedpick
    • First observedrank

Related MCP Connectors

Related MCP Servers

  • ai-compass

    kalraakshit042
    A
    license
    D
    quality
    D
    maintenance
    Describe your AI use case in plain English, get ranked model recommendations with cost estimates and tradeoff reasoning. Covers 62 models across 29 providers. Available as a web app (BYOK + guest tier) and as an MCP server for Claude Desktop and Cursor — same recommendation engine, two interfaces.
    1
    1
    MIT

  • Index9 MCP Server

    index9-org
    A
    license
    A
    quality
    F
    maintenance
    Provides AI assistants with real-time access to 1000+ AI models including their latest pricing, context windows, capabilities, and specifications. Supports model search, comparison, recommendations, and live testing.
    3
    104 npm
    1
    MIT
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources