XFMS picks the right LLM model for any stated task. You give it a concrete purpose ("fixing bugs in a Python codebase", "summarizing 50-page commercial leases"), and it infers which quality benchmarks matter, weighs every model in its catalog against those dimensions, and returns a ranked shortlist with plain-English rationale per pick.
The catalog updates continuously from 8 independent third-party evaluators — no provider self-reports, no single-source benchmarks.
Scored across 5 tools
Each tool has a distinct purpose: discover explains dimensions without ranking, pick returns one answer, rank returns a shortlist, compare tests user-specified models, and benchmark tests the engine's own top picks. The two A/B tools are explicitly differentiated by whether the user or the engine selects candidates, with clear guidance on when to use each.
All five tools follow a consistent single-word lowercase verb pattern: discover, pick, rank, compare, benchmark. The naming is uniform, intuitive, and matches each tool's action clearly.
Five tools is well-scoped for the domain of LLM selection and evaluation. Each tool covers a distinct phase of the workflow—understanding, choosing, ranking, and live testing—without redundancy or unnecessary bulk.
The tool set covers the full decision-making workflow: discover provides pre-ranking insight, pick gives a single recommendation, rank gives alternatives, compare tests user-specified models, and benchmark stress-tests engine-chosen candidates. There are no obvious dead ends or missing operations for the stated purpose.
Run a live A/B test against the engine's TOP 3 PICKS for a stated purpose — the engine chooses the candidates from the full catalog. Generates 5 representative test queries (auto-expands to 10 or 15 if results are too close to call), runs them through the picked models in parallel, and returns real cost, latency, and plain-English commentary on who won what. Use AFTER pick or rank when the user wants the engine's own picks stress-tested with live data. DO NOT use this when the user has already named specific candidate models — the engine will ignore the names and test its own picks. Use compare instead in that case. Costs more than rank (15+ live LLM calls).
| Name | Required | Description | Default |
|---|---|---|---|
| top_n | No | How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives. | |
| primary | No | Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner. | |
| purpose | Yes | One sentence describing what the model will be used for. The benchmark generates representative test queries from this — so be concrete, not vague. | |
| capabilities | No | Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use. | |
| test_queries | No | Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples. |
| Name | Required | Description |
|---|---|---|
| models | No | Ranked shortlist of models, highest score first. |
| status | No | 'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered). |
| ab_result | No | |
| catalog_size | No | |
| filtered_out | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
| quality_floor_reason | No | |
| frontier_filtered_out | No | How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint/openWorldHint/idempotentHint, and the description adds substantial behavioral context BEYOND those hints. It discloses that the engine silently ignores user-named candidate models, that test queries auto-expand from 5 to 10 or 15 when results are close, that models run in parallel, and that the operation costs 15+ live LLM calls. This is exactly the kind of behavioral disclosure that helps an agent invoke it safely.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is about five sentences but packs only high-value information: main behavior, auto-expansion behavior, return values, when-to-use, when-not-to-use, and cost. No filler or repetition of schema details. The usage guidance is front-loaded after the core behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with 100% schema coverage and an output schema, the description is complete: it covers candidate selection, dynamic test query generation, return payload (cost, latency, commentary), and placement among siblings. An agent has everything needed to decide whether to call it and what to expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the structured schema already documents all five parameters. The description adds some behavioral context (e.g., that purpose drives generated queries and that test_queries can be supplied) but does not add syntax/type/enum meaning beyond the schema. Per the calibration baseline, 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Run a live A/B test against the engine's TOP 3 PICKS for a stated purpose.' It also names the key output (cost, latency, commentary) and distinguishes from siblings by naming compare/pick/rank placement. An agent can tell it apart from compare without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says 'Use AFTER `pick` or `rank` when the user wants the engine's own picks stress-tested with live data.' It also gives a negative rule: 'DO NOT use this when the user has already named specific candidate models — the engine will ignore the names and test its own picks. Use `compare` instead in that case.' It even adds cost guidance ('Costs more than `rank`'). Nothing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a live A/B test between 2–5 user-specified models for a stated purpose. NO ranking step — the supplied model_ids ARE the candidate set. Generates 5 representative test queries from the purpose, runs them through every named model in parallel, and returns real cost, latency, and plain-English commentary on who won what. Unknown IDs are dropped with a note; if fewer than 2 IDs resolve, the call refuses. Use this whenever the user names specific models to compare (e.g. 'A/B test X and Y'). For engine-chosen candidates, use benchmark instead. Costs more than rank (10+ live LLM calls). Free-tier note: when any candidate ends in ':free', the probe is capped at 3 queries (no adaptive expansion) because free-tier rate limits often push longer probes past the deploy's 5-minute ceiling — evidence will be shallower. The commentary surfaces this when it happens.
| Name | Required | Description | Default |
|---|---|---|---|
| primary | No | Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner. | |
| purpose | Yes | One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague. | |
| model_ids | Yes | Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case. | |
| test_queries | No | Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples. |
| Name | Required | Description |
|---|---|---|
| status | No | |
| purpose | No | |
| ab_result | No | |
| refusal_reason | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
| model_ids_tested | No | |
| invalid_model_ids | No | |
| model_ids_requested | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations (read-only, idempotent, non-destructive), it discloses that the tool generates 5 representative queries, runs them in parallel, drops unknown IDs, refuses when fewer than 2 resolve, costs 10+ live LLM calls, and changes behavior for ':free' models (capped at 3 queries). This is substantial behavioral context that annotations alone do not convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the primary behavior and alternatives before edge-case details. It is longer than strictly necessary and repeats some schema content (unknown ID dropping, refusal threshold, do-not-call-benchmark), but the repeated points are important enough that the prose remains efficient overall.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the rich annotations and 100% schema coverage, the description completes the picture with failure behavior, cost implications, free-tier limitations, and explicit sibling-tool routing. An agent has everything needed to decide whether and how to invoke compare correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds behavioral reinforcement for model_ids and purpose (e.g., 'NO ranking step', '5 representative test queries'), but it does not explain `primary` or `test_queries` beyond what the schema already provides. It adds no new parameter syntax or format details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Run a live A/B test between 2–5 user-specified models for a stated purpose.' It explicitly disambiguates from siblings by saying there is 'NO ranking step' and that the supplied model_ids ARE the candidate set, and it names what it returns (cost, latency, commentary). An agent can understand what it does immediately and distinguish it from benchmark/rank.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit routing: 'Use this whenever the user names specific models to compare' and 'For engine-chosen candidates, use benchmark instead.' The description also warns that compare is more expensive than rank, giving a cost-based consideration for choosing between tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Show which quality dimensions matter for a stated purpose, WITHOUT ranking any models. Returns the inferred weights and the discovery-walk trace. Useful for understanding how XFMS interprets the purpose before committing to a pick.
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing the task. The tool returns which quality dimensions XFMS would weigh for this purpose, without actually ranking any models. Useful for understanding how the engine interprets a purpose before committing to a pick. |
| Name | Required | Description |
|---|---|---|
| events | No | Trace of the discovery walk. |
| weights | No | Per-dimension weights inferred for this purpose. |
| derived_purpose | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld, and non-destructive behavior. The description adds value beyond these by revealing the output specifics: 'Returns the inferred weights and the discovery-walk trace,' and reinforces the non-ranking behavior. This provides useful additional context without contradicting annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the primary action and key caveat, and every clause adds value. It avoids redundancy and conveys the essential facts efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With a single parameter, a rich output schema, and comprehensive annotations, the description covers the tool's purpose, behavior, and usage context. It explains what the tool does, what it returns, and when to use it, leaving no significant gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the parameter description in the schema is already detailed, explaining both the input and its purpose. The tool description adds no additional semantics for the parameter beyond what the schema states, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Show') and resource ('quality dimensions'), clearly stating the tool's function. It explicitly says 'WITHOUT ranking any models,' which differentiates it from sibling tools like rank and pick, making its unique purpose unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear when-to-use context: 'Useful for understanding how XFMS interprets the purpose before committing to a pick.' It also states a when-not-to-use condition ('WITHOUT ranking any models'). However, it does not explicitly name alternative tools for ranking, instead implying 'pick' as a subsequent step, so it stops short of fully explicit alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the single best LLM for a stated purpose. Concise output, no list. Use when the user has settled on the criteria and just wants one answer.
| Name | Required | Description | Default |
|---|---|---|---|
| purpose | Yes | One sentence describing what the model will be used for. Be concrete, not vague: 'summarizing 50-page commercial leases' works; 'summarization' does not. |
| Name | Required | Description |
|---|---|---|
| name | No | |
| model_id | No | |
| provider | No | |
| rationale | No | |
| total_score | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, non-destructive, and open-world traits, so the bar is lower. The description adds useful behavioral context beyond those: output is concise, returns exactly one LLM, and returns no list. It does not explain how the model decides 'best' or note potential subjectivity, but for a low-risk read-only tool this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with zero filler. The core behavior ('Return the single best LLM') is front-loaded, and the usage condition follows immediately. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a single well-documented parameter, rich annotations, and an output schema, the description covers what an agent needs: what it does, when to use it, and what form the answer takes. Nothing essential is missing for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, purpose, has 100% schema description coverage with concrete guidance and examples. The description echoes 'stated purpose' but adds no new semantic detail beyond the schema. Baseline 3 is appropriate because the schema carries the parameter meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Return the single best LLM for a stated purpose.' It further distinguishes the tool by emphasizing 'single best' and 'no list,' which separates it from siblings like rank or compare. This is a clear, non-tautological purpose statement.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an explicit usage condition: 'Use when the user has settled on the criteria and just wants one answer.' This provides clear context for when to invoke it, though it does not explicitly name alternatives or state when not to use it. The 'no list' phrasing indirectly hints at exclusion of ranking-style outputs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Rank LLMs for a stated purpose. Returns a shortlist with weights, scores, and plain-English rationale per pick. Use when the user wants to see and compare alternatives, not just one answer.
| Name | Required | Description | Default |
|---|---|---|---|
| top_n | No | How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives. | |
| primary | No | Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner. | |
| purpose | Yes | One sentence describing what the model will be used for. Be concrete, not vague: 'fixing bugs in a Python codebase' works; 'coding' does not. The more specific the purpose, the better XFMS can infer which quality dimensions matter. | |
| capabilities | No | Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use. |
| Name | Required | Description |
|---|---|---|
| models | No | Ranked shortlist of models, highest score first. |
| status | No | 'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered). |
| catalog_size | No | |
| filtered_out | No | |
| xpansion_update | No | Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates. |
| quality_floor_reason | No | |
| frontier_filtered_out | No | How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover the safety profile (readOnly, idempotent, openWorld, non-destructive), and the description adds useful behavioral context beyond them: it returns multiple candidates with weights, scores, and human-readable rationale, implying a multi-answer ranked presentation. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place: action, output shape, and usage condition. The core verb is front-loaded, and there is zero filler or redundant restatement of schema content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a 100%-covered schema, a present output schema, and rich annotations, the description covers the essential when/what. The only notable omission is guidance on how ranking decisions (weights/scores) should be interpreted or how this differs from compare-style analysis, but the structured fields already carry most of that burden.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3; the schema itself documents all four parameters well. The description adds little parameter-specific meaning beyond 'stated purpose' scaffolding — it does not compensate with extra semantics, but it doesn't need to given the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('Rank LLMs') and describes a distinctive output (shortlist with weights, scores, plain-English rationale). The phrase 'not just one answer' separates it from the 'pick' sibling, but it does not explicitly distinguish it from 'compare', 'benchmark', or 'discover', so sibling differentiation is partial.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives a clear trigger condition: 'Use when the user wants to see and compare alternatives, not just one answer.' This provides solid when-to-use context but includes no exclusions and does not name alternative tools or route the agent to them, as required for a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
benchmark5 fields changedInput schema / properties / capabilitiesAdded value: +{
+ "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
+ "items": {
+ "enum": [
+ "vision",
+ "audio_in",
+ "tool_use",
+ "structured_outputs"
+ ],
+ "type": "string"
+ },
+ "type": "array"
+}Input schema / properties / primaryAdded value: +{
+ "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
+ "items": {
+ "enum": [
+ "cost",
+ "quality",
+ "latency",
+ "privacy"
+ ],
+ "type": "string"
+ },
+ "type": "array"
+}Input schema / properties / test_queriesAdded value: +{
+ "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+ "items": {
+ "minLength": 1,
+ "type": "string"
+ },
+ "maxItems": 15,
+ "minItems": 1,
+ "type": "array"
+}Input schema / properties / top_nAdded value: +{
+ "default": 5,
+ "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
+ "maximum": 25,
+ "minimum": 1,
+ "type": "integer"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / typePrevious value: -"number"New value: +[
+ "number",
+ "null"
+]compare3 fields changedInput schema / properties / primary / descriptionPrevious value: -"Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."Input schema / properties / test_queriesAdded value: +{
+ "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+ "items": {
+ "minLength": 1,
+ "type": "string"
+ },
+ "maxItems": 15,
+ "minItems": 1,
+ "type": "array"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / typePrevious value: -"number"New value: +[
+ "number",
+ "null"
+]rank1 field changedInput schema / properties / primary / descriptionPrevious value: -"Mark dimensions as primary tier. When set, the engine switches from weighted-sum blending to lexicographic ordering: the primary dimension is the sole ranking axis, and other dimensions only break ties. Use when the user says 'cheapest model, period' or similar — their stated preference becomes sacrosanct."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."benchmark5 fields changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracyAdded value: +{
+ "type": "number"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / runsAdded value: +{
+ "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+ "items": {
+ "properties": {
+ "error": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "response_text": {
+ "type": "string"
+ },
+ "test_query": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "type": "array"
+}Output schema / properties / frontier_filtered_outAdded value: +{
+ "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+ "type": "integer"
+}Output schema / properties / quality_floor_reasonAdded value: +{
+ "type": [
+ "string",
+ "null"
+ ]
+}Output schema / properties / status / descriptionAdded value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."compare2 fields changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracyAdded value: +{
+ "type": "number"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / runsAdded value: +{
+ "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+ "items": {
+ "properties": {
+ "error": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "response_text": {
+ "type": "string"
+ },
+ "test_query": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "type": "array"
+}pick1 field changedOutput schema / descriptionPrevious value: -"The single best model. Same shape as rank's models[0]."New value: +"The single best model — same shape as rank's models[0]. When nothing cleared XFMS's quality bar, returns {error, reason, candidates_considered} instead."rank3 fields changedOutput schema / properties / frontier_filtered_outAdded value: +{
+ "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+ "type": "integer"
+}Output schema / properties / quality_floor_reasonAdded value: +{
+ "type": [
+ "string",
+ "null"
+ ]
+}Output schema / properties / status / descriptionAdded value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."benchmark1 field changedOutput schema / properties / models / items / properties / provider / typePrevious value: -"string"New value: +[
+ "string",
+ "null"
+]pick1 field changedOutput schema / properties / provider / typePrevious value: -"string"New value: +[
+ "string",
+ "null"
+]rank1 field changedOutput schema / properties / models / items / properties / provider / typePrevious value: -"string"New value: +[
+ "string",
+ "null"
+]comparebenchmark1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}discover1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}pick1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}rank1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}benchmarkdiscoverpickrank
Add one secure layer between your agents and this server.