XFMS picks the right LLM model for any stated task. You give it a concrete purpose ("fixing bugs in a Python codebase", "summarizing 50-page commercial leases"), and it infers which quality benchmarks matter, weighs every model in its catalog against those dimensions, and returns a ranked shortlist with plain-English rationale per pick.
The catalog updates continuously from 8 independent third-party evaluators — no provider self-reports, no single-source benchmarks.
Scored across 7 tools
benchmark and compare are near-identical live-test flows differing only in where the model list comes from; rank and pick both consume a discover session and return overlapping shortlist data. field_results, discover, and watch_job are clearly distinct, and the descriptions do explain the differences, but the boundaries remain soft enough to invite misselection.
Most tools are bare imperative verbs (benchmark, compare, discover, pick, rank) while watch_job is verb_noun and field_results is a noun phrase. It is readable but there is no single predictable pattern to infer a new tool's name.
Seven tools map cleanly onto the stages of model selection: discover requirements, rank/pick, run live tests, consult field data, and monitor. The only redundancy is the benchmark/compare pair, which is a minor over-scoping rather than bloat.
The surface covers the full lifecycle from requirement discovery through ranking, live benchmarking, and ongoing re-testing via Model Watch, plus aggregate field_results. Gaps are modest: no tool to list or re-fetch prior sessions/reports, and no account or subscription management despite watch_job requiring a signed-in subscriber.
The live test: the 5 proven models and 1 promising newcomer from discovery do the person's real tasks side by side, every requirement is judged, and the results pick the winner. First call (session + tasks) returns the estimate and an offer — nothing is spent. Show the person the cost; only when they agree, call again with offer and go=true. Runs on the person's OpenRouter key (X-OpenRouter-Key header). Save the returned report as a local file.
| Name | Required | Description | Default |
|---|---|---|---|
| go | No | The person approved the estimate. Spends on their key. | |
| offer | No | The offer from the estimate. Pass it with go=true to run. | |
| tasks | No | The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs. | |
| ratings | No | Blind ratings: {task id: {letter: good|ok|poor}}, with test_session. | |
| session | Yes | The sealed session from a finished discover conversation. | |
| overruled | No | Failures the person accepts, as 'model id|requirement words'. | |
| jobs_wanted | No | How many AI jobs the person would want watched, if they said; sizes the Model Watch plan. | |
| test_session | No | From a finished run, to overrule failures without rerunning. | |
| current_model | No | OpenRouter id of the model the job uses now, if any. | |
| runs_per_month | No | Roughly how many times a month the job runs, if known. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With annotations already marking it read-only, non-destructive, and open-world, the description adds important behavioral context: no spend on the first call, approval gating before spending, use of the X-OpenRouter-Key header, and saving the returned report. It does not clarify duplicate-spend/idempotency behavior when go=true is repeated or rate limits, so it is not fully complete.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four tightly packed sentences front-load the live-test purpose, then the two-step spending protocol and key/report details. Every sentence carries operational information, with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists and input schema descriptions are complete, so the description need not explain return values. It covers the crucial two-phase invocation, cost approval, authentication header, and report handling, making it complete enough for an agent to call safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the schema already documents go, offer, tasks, ratings, session, overruled, and the other parameters. The description usefully links session/tasks to the estimate call and offer/go to the paid call, but adds little parameter-level meaning beyond the existing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a concrete action: run a live side-by-side test of the shortlisted models on the person's real tasks, judge every requirement, and pick a winner. The phrase 'from discovery' and '5 proven models and 1 promising newcomer' ties it to the pipeline and separates it from sibling discovery/pick/rank tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It spells out a two-call protocol: first call with session and tasks returns an estimate/offer and spends nothing; after showing the cost and getting agreement, call again with offer and go=true. This is clear when to invoke and when not to spend, though it does not discuss sibling alternatives such as compare or rank.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Same live test as benchmark, but on the models the person names (model_ids). Needs a finished discover session so every requirement is judged. Estimate first, then go=true.
| Name | Required | Description | Default |
|---|---|---|---|
| go | No | The person approved the estimate. Spends on their key. | |
| offer | No | The offer from the estimate. Pass it with go=true to run. | |
| tasks | No | The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs. | |
| ratings | No | Blind ratings: {task id: {letter: good|ok|poor}}, with test_session. | |
| session | Yes | The sealed session from a finished discover conversation. | |
| model_ids | No | The OpenRouter ids of the models the person named. | |
| overruled | No | Failures the person accepts, as 'model id|requirement words'. | |
| test_session | No | From a finished run, to overrule failures without rerunning. | |
| current_model | No | OpenRouter id of the model the job uses now, if any. | |
| runs_per_month | No | Roughly how many times a month the job runs, if known. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description frames a live, stateful test whose second phase ('then go=true') consumes the person's credits and creates a run/session — yet the annotations declare readOnlyHint=true and idempotentHint=true. Labeling a spend-bearing, non-idempotent execution as read-only/idempotent is a contradiction that could cause a client to auto-approve a paid run.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with what the tool is, then the prerequisite, then the action flow. No filler; every clause carries information, though it spends no words on the cost implication it arguably should surface.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool (10 params, nested objects, rich schema, output schema present), the description covers purpose, prerequisite, workflow and sibling relationship; return values are unnecessary given the output schema. The only meaningful omission — the paid/gated nature of go=true — is documented in the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all 10 parameters (including nested task fields and the go/offer gate) are already documented in the schema. The description only echoes model_ids and the estimate/go flow, adding no syntax or semantics beyond the schema — the baseline 3 for full coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource — running a live test of the person's named models on their own tasks — and explicitly contrasts with the sibling: 'Same live test as benchmark, but on the models the person names (model_ids).' An agent can tell compare and benchmark apart without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a hard prerequisite ('Needs a finished discover session so every requirement is judged') and the sequencing ('Estimate first, then go=true'), which routes the agent to discover/benchmark appropriately. It stops short of explicit when-not-to-use or cost caveats, but the context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Discovery: finds the true requirements of the person's job before any model is chosen. Start with purpose: the reply carries the discovery instructions and the evidence catalogue. YOU ask the person the questions (one thing each, plain words, offering the returned choices); then call again with the sealed session, their answers and your structured updates. Each reply gives progress, rejected updates with reasons, next targets, answer choices from live data and a shortlist. At most 5 rounds. Ends with a confirmation view; call with confirm=true once the person agrees. The person can stop any time (end_now=true). Nothing about the person is stored by XFMS.
| Name | Required | Description | Default |
|---|---|---|---|
| answers | No | The person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words. For an XFMS question the person already answered earlier, send {"choice": choice id, "from_earlier": their exact earlier words} instead of asking it. | |
| confirm | No | The person agrees with the confirmation view. | |
| context | No | Optional, with the person's consent: short labelled excerpts (sample inputs, a conversation summary). Used for this request only, never kept. | |
| end_now | No | The person wants to stop and see the best result so far. | |
| purpose | No | The job the person wants an AI model for, in their own words. Needed to start. | |
| session | No | The sealed session value from the previous discover response. Pass it back unchanged. | |
| updates | No | What you learned from the answers, in the format the first discover reply's instructions give: conclusions, requirements, shapes, open_items, item_moves, relation, usage, survivors_matter, pushback. XFMS checks each against its rules and reports any it rejects. | |
| corrections | No | Changes the person makes in the confirmation view: {requirement_id, field, value}. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnly, idempotent, non-destructive, openWorld), so the bar is lower, and the description still adds substantial protocol behavior: the 5-round cap, the confirmation-view step, the stop escape hatch, and the no-retention guarantee ('Nothing about the person is stored by XFMS'). What it does not disclose is what a mid-protocol failure or session expiry does to accumulated state.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core purpose ('Discovery: finds the true requirements...') and every sentence carries protocol information for an otherwise 8-parameter stateful tool. It is a dense single paragraph rather than a scannable protocol outline, which costs it the top score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return contents need not be explained, yet the description still sketches what each reply carries (progress, rejected updates, next targets, choices, shortlist) and defers the detailed update grammar to runtime instructions. For a multi-round tool this is nearly complete; only failure/recovery behavior for the sealed session is left unstated.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so per-parameter meaning is already handled and the baseline is 3. The description adds real sequencing semantics the schema cannot convey: purpose starts the flow, session is the sealed token passed back unchanged, confirm/end_now are the two exit gates, and updates must match the previously returned instruction format.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: it 'finds the true requirements of the person's job before any model is chosen,' which is a genuine elicitation capability rather than a restatement of the name. It implicitly differentiates itself from the downstream siblings (pick, rank, compare, benchmark) by positioning itself as the pre-model step, though it never names one.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear entry conditions ('Start with purpose'), the continuation condition (call again with the sealed session, answers and updates), and the termination paths (confirm=true once agreed, end_now=true to stop, max 5 rounds). It establishes this is the first step of the pipeline but does not explicitly name which sibling to call next.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Results from real comparisons on Model Source for one kind of job (call with no job_type for the list): which models passed, how often, which won, typical cost per task and the most common failure. Counted without anyone's job details. A starting point; test on the person's own work before choosing.
| Name | Required | Description | Default |
|---|---|---|---|
| job_type | No | One key from the list returned when called without it. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, non-destructive behavior, so the bar is lower, and the description still adds meaningful context: results are anonymized aggregates ('Counted without anyone's job details') and are advisory rather than decision-grade. It does not discuss staleness, sample size, or pagination, which would have pushed this higher.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and payload are front-loaded in the opening clause, and the closing caveat is short and earns its place. The first sentence is long and comma-heavy, which slightly hurts scanability but wastes no words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the burden of describing return values and does so explicitly (pass rates, winners, cost per task, common failure), plus it explains the empty-argument case. What is missing is any indication of how much data or how recent the underlying comparisons are.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the single optional parameter, so the baseline is 3. The description restates the no-argument behavior (get the list of valid job_type values) that the schema already documents, adding no new syntax or format detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific resource (aggregated field comparison results on Model Source) and enumerates the payload (passes, win rates, cost per task, common failure), so an agent knows exactly what it returns. It does not, however, distinguish itself from siblings like benchmark or rank, leaving that differentiation to inference.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives real usage context: call with no job_type to get the available list, treat the output as a starting point, and validate on the person's own work before choosing. That is implied guidance rather than explicit when-to-use/when-not rules, and no sibling alternative is named for cases where this data is insufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The single best proven model for a job worked out with discover, with why and the runners-up. Pass the discover session.
| Name | Required | Description | Default |
|---|---|---|---|
| session | Yes | The sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, non-destructive and openWorld, so the safety profile is covered. The description adds that the result is singular and includes 'why and the runners-up,' but that overlaps with the existing output schema and adds little behavioral context beyond it.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with no waste; the core purpose is front-loaded and the passing instruction closes it. Slightly telegraphic phrasing ('worked out with discover') costs a little clarity but nothing is padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, return values need not be spelled out, and annotations carry the safety profile. What remains is adequate for a one-parameter tool; only the relationship to overlapping siblings such as rank and compare is left unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single 'session' parameter, and the schema description already explains it is the sealed discover session. The description's 'Pass the discover session' merely restates the schema rather than adding format, source, or validation detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: returns the single best proven model for a job plus why and runners-up. It references the discover workflow, tying it to a sibling, but does not explicitly distinguish itself from rank or compare, which overlap in the ranking space.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The prerequisite is explicit: the job must be 'worked out with discover' and you must 'Pass the discover session,' reinforced by the schema's 'call discover first.' It gives clear context but no exclusions or comparison against siblings like rank or compare.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The full shortlist for a job worked out with discover: 5 proven models and 1 promising newcomer, each with its standing on every requirement, plus near misses and what would change the order. Pass the discover session.
| Name | Required | Description | Default |
|---|---|---|---|
| session | Yes | The sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, non-destructive, open-world, so the safety profile is fully covered and the description need not restate it. It does add one genuine behavioral trait beyond the annotations — the deterministic composition (5 proven models + 1 newcomer) and that the order is sensitivity-aware ("what would change the order") — but the rest is return-value description that the output schema already carries.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence that leads with the deliverable and then enumerates its parts. Dense but every clause maps to real content; no preamble or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The prerequisite (run discover first) and the shape of the result are both conveyed, and an output schema exists so return values need not be spelled out further. The only real gap is the absence of any routing guidance against the sibling tools that also consume discovery output.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is a single parameter with 100% schema description coverage, and the schema already explains it is a sealed session from a discover conversation. The description's "Pass the discover session" merely echoes that, adding no format or validity detail beyond the schema, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names the resource (models for a discovered job) and the artifact produced (the full shortlist), and ties itself to the discover flow so an agent can distinguish it from siblings like discover or benchmark. It describes the output more than the act of ranking, so the verb is implied rather than stated, but the intent is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Pass the discover session" plus the schema note that discovery must run first gives a clear prerequisite, but there is no explicit statement of when to choose rank over pick, compare, or benchmark. Usage is implied by the discover dependency rather than contrasted with alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Saves a finished job for XF Model Watch, which re-tests it as new models arrive and emails the person only when a model passes every must-have and ranks higher on their priorities. Needs a signed-in subscriber. Ask the person first: Model Watch keeps the job and its example tasks (encrypted, deletable any time). Pass consent=true only if they agree. Pass the discover session, and either test_session from a finished comparison or the tasks, plus the model they use now. The first check runs as soon as the job is saved. To re-check a job the person already watches, pass check_now with the job's id or title instead (once a day per job; the result is emailed and shown at /account).
| Name | Required | Description | Default |
|---|---|---|---|
| tasks | No | Example tasks, if no test_session. | |
| title | No | A short name for the job, in the person's words. | |
| consent | No | The person agreed to Model Watch keeping this job. | |
| session | No | The discover session. | |
| check_now | No | Check a job they already watch now: its id or title. Leave out when saving a job. | |
| test_session | No | From a finished comparison; carries its tasks. | |
| current_model | No | OpenRouter id of the model the job uses now. | |
| runs_per_month | No | Roughly how many times a month the job runs, if known. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=false, destructiveHint=false, and idempotentHint=false. The description adds substantial context beyond annotations: encryption and deletability of stored data, consent requirement, immediate first check, once-a-day rate limit for re-checks, and that results are emailed and shown at /account.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the purpose in the first sentence, then follows with prerequisites, required inputs, timing, and the re-check alternative. Every sentence earns its place for an 8-parameter tool with complex consent and storage behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Although there is no output schema, the description explains where results appear (emailed and shown at /account) and covers all prerequisites, consent handling, storage traits, and timing. Annotations carry the safety profile, and the description fills in the remaining behavioral gaps completely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds some meaning beyond the schema, such as passing consent=true only if the person agrees and clarifying the either/or relationship between test_session and tasks. However, most parameter semantics are already in the schema descriptions, so it does not reach the highest level.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Saves a finished job for XF Model Watch' and explains the monitoring behavior (re-tests, emails when a model ranks higher). This distinguishes it from siblings like benchmark, compare, and discover. Also covers the secondary re-check mode explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use it: after a finished job, needs a signed-in subscriber, ask the person first, pass consent=true only if they agree. Gives the alternative for re-checking: 'To re-check a job the person already watches, pass check_now with the job's id or title instead.' Nothing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
benchmark1 field changedInput schema / properties / jobs_wantedAdded value: +{
+ "description": "How many AI jobs the person would want watched, if they said; sizes the Model Watch plan.",
+ "type": "integer"
+}watch_job2 fields changedInput schema / properties / check_nowAdded value: +{
+ "description": "Check a job they already watch now: its id or title. Leave out when saving a job.",
+ "type": [
+ "integer",
+ "string"
+ ]
+}Input schema / requiredPrevious value: -[
- "session",
- "consent"
-]New value: +[]watch_job1 field changedInput schema / properties / consent / descriptionPrevious value: -"The person agreed to Watch keeping this job."New value: +"The person agreed to Model Watch keeping this job."benchmark2 fields changedInput schema / properties / current_modelAdded value: +{
+ "description": "OpenRouter id of the model the job uses now, if any.",
+ "type": "string"
+}Input schema / properties / runs_per_monthAdded value: +{
+ "description": "Roughly how many times a month the job runs, if known.",
+ "type": "number"
+}compare2 fields changedInput schema / properties / current_modelAdded value: +{
+ "description": "OpenRouter id of the model the job uses now, if any.",
+ "type": "string"
+}Input schema / properties / runs_per_monthAdded value: +{
+ "description": "Roughly how many times a month the job runs, if known.",
+ "type": "number"
+}watch_jobdiscover3 fields changedInput schema / properties / answers / additionalProperties / anyOfAdded value: +[
+ {
+ "type": "string"
+ },
+ {
+ "properties": {
+ "choice": {
+ "type": "string"
+ },
+ "from_earlier": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "choice",
+ "from_earlier"
+ ],
+ "type": "object"
+ }
+]Input schema / properties / answers / additionalProperties / typeRemoved value: -"string"Input schema / properties / answers / descriptionPrevious value: -"The person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words."New value: +"The person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words. For an XFMS question the person already answered earlier, send {\"choice\": choice id, \"from_earlier\": their exact earlier words} instead of asking it."field_resultsbenchmark16 fields changedInput schema / properties / capabilitiesRemoved value: -{
- "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
- "items": {
- "enum": [
- "vision",
- "audio_in",
- "tool_use",
- "structured_outputs"
- ],
- "type": "string"
- },
- "type": "array"
-}Input schema / properties / goAdded value: +{
+ "description": "The person approved the estimate. Spends on their key.",
+ "type": "boolean"
+}Input schema / properties / offerAdded value: +{
+ "description": "The offer from the estimate. Pass it with go=true to run.",
+ "type": "string"
+}Input schema / properties / overruledAdded value: +{
+ "description": "Failures the person accepts, as 'model id|requirement words'.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+}Input schema / properties / primaryRemoved value: -{
- "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
- "items": {
- "enum": [
- "cost",
- "quality",
- "latency",
- "privacy"
- ],
- "type": "string"
- },
- "type": "array"
-}Input schema / properties / purposeRemoved value: -{
- "description": "One sentence describing what the model will be used for. The benchmark generates representative test queries from this — so be concrete, not vague.",
- "type": "string"
-}Input schema / properties / ratingsAdded value: +{
+ "description": "Blind ratings: {task id: {letter: good|ok|poor}}, with test_session.",
+ "type": "object"
+}Input schema / properties / sessionAdded value: +{
+ "description": "The sealed session from a finished discover conversation.",
+ "type": "string"
+}Input schema / properties / tasksAdded value: +{
+ "description": "The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.",
+ "items": {
+ "properties": {
+ "covers": {
+ "description": "ids of the requirements this task tests",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "group": {
+ "description": "optional kind of task, e.g. 'short captions'",
+ "type": "string"
+ },
+ "image_urls": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "source": {
+ "description": "real | made up",
+ "type": "string"
+ },
+ "text": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "type": "array"
+}Input schema / properties / test_queriesRemoved value: -{
- "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
- "items": {
- "minLength": 1,
- "type": "string"
- },
- "maxItems": 15,
- "minItems": 1,
- "type": "array"
-}Input schema / properties / test_sessionAdded value: +{
+ "description": "From a finished run, to overrule failures without rerunning.",
+ "type": "string"
+}Input schema / properties / top_nRemoved value: -{
- "default": 5,
- "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
- "maximum": 25,
- "minimum": 1,
- "type": "integer"
-}Input schema / requiredPrevious value: -[
- "purpose"
-]New value: +[
+ "session"
+]Output schema / additionalPropertiesAdded value: +trueOutput schema / descriptionPrevious value: -"Rank response with ab_result populated — same shape as `rank` plus live performance data from the probe runs."New value: +"Without go: the estimate and offer. With go: the outcome, every model on every requirement, the report."Output schema / propertiesRemoved value: -{
- "ab_result": {
- "properties": {
- "aggregates": {
- "description": "Per-model stats across the runs.",
- "items": {
- "properties": {
- "avg_accuracy": {
- "type": [
- "number",
- "null"
- ]
- },
- "avg_completion_tokens": {
- "type": "number"
- },
- "avg_latency_ms": {
- "type": "number"
- },
- "model_id": {
- "type": "string"
- },
- "model_name": {
- "type": "string"
- },
- "runs": {
- "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
- "items": {
- "properties": {
- "error": {
- "type": [
- "string",
- "null"
- ]
- },
- "response_text": {
- "type": "string"
- },
- "test_query": {
- "type": "string"
- }
- },
- "type": "object"
- },
- "type": "array"
- },
- "success_count": {
- "type": "integer"
- },
- "total_cost_usd": {
- "type": "number"
- }
- },
- "type": "object"
- },
- "type": "array"
- },
- "commentary": {
- "type": "string"
- },
- "cost_winner_id": {
- "type": [
- "string",
- "null"
- ]
- },
- "incongruity_detected": {
- "type": "boolean"
- },
- "latency_winner_id": {
- "type": [
- "string",
- "null"
- ]
- },
- "overall_winner_id": {
- "type": [
- "string",
- "null"
- ]
- },
- "queries_executed": {
- "type": "integer"
- },
- "test_queries": {
- "description": "The generated test prompts that were run.",
- "items": {
- "type": "string"
- },
- "type": "array"
- }
- },
- "type": "object"
- },
- "catalog_size": {
- "type": "integer"
- },
- "filtered_out": {
- "type": "integer"
- },
- "frontier_filtered_out": {
- "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
- "type": "integer"
- },
- "models": {
- "description": "Ranked shortlist of models, highest score first.",
- "items": {
- "properties": {
- "model_id": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "provider": {
- "type": [
- "string",
- "null"
- ]
- },
- "rationale": {
- "type": "string"
- },
- "total_score": {
- "type": "number"
- }
- },
- "type": "object"
- },
- "type": "array"
- },
- "quality_floor_reason": {
- "type": [
- "string",
- "null"
- ]
- },
- "status": {
- "description": "'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered).",
- "type": "string"
- },
- "xpansion_update": {
- "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
- "properties": {
- "call_count": {
- "type": "integer"
- },
- "id": {
- "type": "string"
- },
- "message": {
- "type": "string"
- },
- "signup_url": {
- "type": "string"
- }
- },
- "type": "object"
- }
-}compare16 fields changedInput schema / properties / goAdded value: +{
+ "description": "The person approved the estimate. Spends on their key.",
+ "type": "boolean"
+}Input schema / properties / model_ids / descriptionPrevious value: -"Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case."New value: +"The OpenRouter ids of the models the person named."Input schema / properties / model_ids / maxItemsRemoved value: -5Input schema / properties / offerAdded value: +{
+ "description": "The offer from the estimate. Pass it with go=true to run.",
+ "type": "string"
+}Input schema / properties / overruledAdded value: +{
+ "description": "Failures the person accepts, as 'model id|requirement words'.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+}Input schema / properties / primaryRemoved value: -{
- "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
- "items": {
- "enum": [
- "cost",
- "quality",
- "latency",
- "privacy"
- ],
- "type": "string"
- },
- "type": "array"
-}Input schema / properties / purposeRemoved value: -{
- "description": "One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague.",
- "type": "string"
-}Input schema / properties / ratingsAdded value: +{
+ "description": "Blind ratings: {task id: {letter: good|ok|poor}}, with test_session.",
+ "type": "object"
+}Input schema / properties / sessionAdded value: +{
+ "description": "The sealed session from a finished discover conversation.",
+ "type": "string"
+}Input schema / properties / tasksAdded value: +{
+ "description": "The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.",
+ "items": {
+ "properties": {
+ "covers": {
+ "description": "ids of the requirements this task tests",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "group": {
+ "description": "optional kind of task, e.g. 'short captions'",
+ "type": "string"
+ },
+ "image_urls": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "source": {
+ "description": "real | made up",
+ "type": "string"
+ },
+ "text": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "type": "array"
+}Input schema / properties / test_queriesRemoved value: -{
- "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
- "items": {
- "minLength": 1,
- "type": "string"
- },
- "maxItems": 15,
- "minItems": 1,
- "type": "array"
-}Input schema / properties / test_sessionAdded value: +{
+ "description": "From a finished run, to overrule failures without rerunning.",
+ "type": "string"
+}Input schema / requiredPrevious value: -[
- "purpose",
- "model_ids"
-]New value: +[
+ "session"
+]Output schema / additionalPropertiesAdded value: +trueOutput schema / descriptionPrevious value: -"Result of a head-to-head A/B between user-named models. NOT a rank response — no ranking happened, so no scores or rationale. Just probe evidence plus a record of which IDs were resolvable."New value: +"Same as benchmark, for the models the person named."Output schema / propertiesRemoved value: -{
- "ab_result": {
- "properties": {
- "aggregates": {
- "description": "Per-model stats across the runs.",
- "items": {
- "properties": {
- "avg_accuracy": {
- "type": [
- "number",
- "null"
- ]
- },
- "avg_completion_tokens": {
- "type": "number"
- },
- "avg_latency_ms": {
- "type": "number"
- },
- "model_id": {
- "type": "string"
- },
- "model_name": {
- "type": "string"
- },
- "runs": {
- "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
- "items": {
- "properties": {
- "error": {
- "type": [
- "string",
- "null"
- ]
- },
- "response_text": {
- "type": "string"
- },
- "test_query": {
- "type": "string"
- }
- },
- "type": "object"
- },
- "type": "array"
- },
- "success_count": {
- "type": "integer"
- },
- "total_cost_usd": {
- "type": "number"
- }
- },
- "type": "object"
- },
- "type": "array"
- },
- "commentary": {
- "type": "string"
- },
- "cost_winner_id": {
- "type": [
- "string",
- "null"
- ]
- },
- "incongruity_detected": {
- "type": "boolean"
- },
- "latency_winner_id": {
- "type": [
- "string",
- "null"
- ]
- },
- "overall_winner_id": {
- "type": [
- "string",
- "null"
- ]
- },
- "queries_executed": {
- "type": "integer"
- },
- "test_queries": {
- "description": "The generated test prompts that were run.",
- "items": {
- "type": "string"
- },
- "type": "array"
- }
- },
- "type": "object"
- },
- "invalid_model_ids": {
- "items": {
- "type": "string"
- },
- "type": "array"
- },
- "model_ids_requested": {
- "items": {
- "type": "string"
- },
- "type": "array"
- },
- "model_ids_tested": {
- "items": {
- "type": "string"
- },
- "type": "array"
- },
- "purpose": {
- "type": "string"
- },
- "refusal_reason": {
- "type": [
- "string",
- "null"
- ]
- },
- "status": {
- "enum": [
- "compared",
- "refused"
- ],
- "type": "string"
- },
- "xpansion_update": {
- "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
- "properties": {
- "call_count": {
- "type": "integer"
- },
- "id": {
- "type": "string"
- },
- "message": {
- "type": "string"
- },
- "signup_url": {
- "type": "string"
- }
- },
- "type": "object"
- }
-}discover12 fields changedInput schema / properties / answersAdded value: +{
+ "additionalProperties": {
+ "type": "string"
+ },
+ "description": "The person's answers this round, keyed by the question you asked them (short key or the question text), in their exact words.",
+ "type": "object"
+}Input schema / properties / confirmAdded value: +{
+ "description": "The person agrees with the confirmation view.",
+ "type": "boolean"
+}Input schema / properties / contextAdded value: +{
+ "description": "Optional, with the person's consent: short labelled excerpts (sample inputs, a conversation summary). Used for this request only, never kept.",
+ "items": {
+ "properties": {
+ "label": {
+ "type": "string"
+ },
+ "text": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "type": "array"
+}Input schema / properties / correctionsAdded value: +{
+ "description": "Changes the person makes in the confirmation view: {requirement_id, field, value}.",
+ "items": {
+ "type": "object"
+ },
+ "type": "array"
+}Input schema / properties / end_nowAdded value: +{
+ "description": "The person wants to stop and see the best result so far.",
+ "type": "boolean"
+}Input schema / properties / purpose / descriptionPrevious value: -"One sentence describing the task. The tool returns which quality dimensions XFMS would weigh for this purpose, without actually ranking any models. Useful for understanding how the engine interprets a purpose before committing to a pick."New value: +"The job the person wants an AI model for, in their own words. Needed to start."Input schema / properties / sessionAdded value: +{
+ "description": "The sealed session value from the previous discover response. Pass it back unchanged.",
+ "type": "string"
+}Input schema / properties / updatesAdded value: +{
+ "description": "What you learned from the answers, in the format the first discover reply's instructions give: conclusions, requirements, shapes, open_items, item_moves, relation, usage, survivors_matter, pushback. XFMS checks each against its rules and reports any it rejects.",
+ "type": "object"
+}Input schema / requiredRemoved value: -[
- "purpose"
-]Output schema / additionalPropertiesAdded value: +trueOutput schema / descriptionAdded value: +"stage (one | two | confirm | confirmed), questions to relay (each with why), open items, coverage, the confirmation view when ready, and the sealed session."Output schema / propertiesRemoved value: -{
- "derived_purpose": {
- "type": "string"
- },
- "events": {
- "description": "Trace of the discovery walk.",
- "type": "array"
- },
- "weights": {
- "description": "Per-dimension weights inferred for this purpose.",
- "type": "object"
- },
- "xpansion_update": {
- "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
- "properties": {
- "call_count": {
- "type": "integer"
- },
- "id": {
- "type": "string"
- },
- "message": {
- "type": "string"
- },
- "signup_url": {
- "type": "string"
- }
- },
- "type": "object"
- }
-}pick6 fields changedInput schema / properties / purposeRemoved value: -{
- "description": "One sentence describing what the model will be used for. Be concrete, not vague: 'summarizing 50-page commercial leases' works; 'summarization' does not.",
- "type": "string"
-}Input schema / properties / sessionAdded value: +{
+ "description": "The sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first.",
+ "type": "string"
+}Input schema / requiredPrevious value: -[
- "purpose"
-]New value: +[
+ "session"
+]Output schema / additionalPropertiesAdded value: +trueOutput schema / descriptionPrevious value: -"The single best model — same shape as rank's models[0]. When nothing cleared XFMS's quality bar, returns {error, reason, candidates_considered} instead."New value: +"The top proven pick with why, plus the runners-up."Output schema / propertiesRemoved value: -{
- "model_id": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "provider": {
- "type": [
- "string",
- "null"
- ]
- },
- "rationale": {
- "type": "string"
- },
- "total_score": {
- "type": "number"
- },
- "xpansion_update": {
- "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
- "properties": {
- "call_count": {
- "type": "integer"
- },
- "id": {
- "type": "string"
- },
- "message": {
- "type": "string"
- },
- "signup_url": {
- "type": "string"
- }
- },
- "type": "object"
- }
-}rank9 fields changedInput schema / properties / capabilitiesRemoved value: -{
- "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
- "items": {
- "enum": [
- "vision",
- "audio_in",
- "tool_use",
- "structured_outputs"
- ],
- "type": "string"
- },
- "type": "array"
-}Input schema / properties / primaryRemoved value: -{
- "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
- "items": {
- "enum": [
- "cost",
- "quality",
- "latency",
- "privacy"
- ],
- "type": "string"
- },
- "type": "array"
-}Input schema / properties / purposeRemoved value: -{
- "description": "One sentence describing what the model will be used for. Be concrete, not vague: 'fixing bugs in a Python codebase' works; 'coding' does not. The more specific the purpose, the better XFMS can infer which quality dimensions matter.",
- "type": "string"
-}Input schema / properties / sessionAdded value: +{
+ "description": "The sealed session from a discover conversation. Ranking is built from what discovery found out about the job — call discover first.",
+ "type": "string"
+}Input schema / properties / top_nRemoved value: -{
- "default": 5,
- "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
- "maximum": 25,
- "minimum": 1,
- "type": "integer"
-}Input schema / requiredPrevious value: -[
- "purpose"
-]New value: +[
+ "session"
+]Output schema / additionalPropertiesAdded value: +trueOutput schema / descriptionAdded value: +"proven picks (each with its standing on every requirement), one promising newcomer, near misses, what changes if priorities change, and the assumptions."Output schema / propertiesRemoved value: -{
- "catalog_size": {
- "type": "integer"
- },
- "filtered_out": {
- "type": "integer"
- },
- "frontier_filtered_out": {
- "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
- "type": "integer"
- },
- "models": {
- "description": "Ranked shortlist of models, highest score first.",
- "items": {
- "properties": {
- "model_id": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "provider": {
- "type": [
- "string",
- "null"
- ]
- },
- "rationale": {
- "type": "string"
- },
- "total_score": {
- "type": "number"
- }
- },
- "type": "object"
- },
- "type": "array"
- },
- "quality_floor_reason": {
- "type": [
- "string",
- "null"
- ]
- },
- "status": {
- "description": "'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered).",
- "type": "string"
- },
- "xpansion_update": {
- "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
- "properties": {
- "call_count": {
- "type": "integer"
- },
- "id": {
- "type": "string"
- },
- "message": {
- "type": "string"
- },
- "signup_url": {
- "type": "string"
- }
- },
- "type": "object"
- }
-}benchmark5 fields changedInput schema / properties / capabilitiesAdded value: +{
+ "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
+ "items": {
+ "enum": [
+ "vision",
+ "audio_in",
+ "tool_use",
+ "structured_outputs"
+ ],
+ "type": "string"
+ },
+ "type": "array"
+}Input schema / properties / primaryAdded value: +{
+ "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
+ "items": {
+ "enum": [
+ "cost",
+ "quality",
+ "latency",
+ "privacy"
+ ],
+ "type": "string"
+ },
+ "type": "array"
+}Input schema / properties / test_queriesAdded value: +{
+ "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+ "items": {
+ "minLength": 1,
+ "type": "string"
+ },
+ "maxItems": 15,
+ "minItems": 1,
+ "type": "array"
+}Input schema / properties / top_nAdded value: +{
+ "default": 5,
+ "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
+ "maximum": 25,
+ "minimum": 1,
+ "type": "integer"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / typePrevious value: -"number"New value: +[
+ "number",
+ "null"
+]compare3 fields changedInput schema / properties / primary / descriptionPrevious value: -"Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."Input schema / properties / test_queriesAdded value: +{
+ "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+ "items": {
+ "minLength": 1,
+ "type": "string"
+ },
+ "maxItems": 15,
+ "minItems": 1,
+ "type": "array"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / typePrevious value: -"number"New value: +[
+ "number",
+ "null"
+]rank1 field changedInput schema / properties / primary / descriptionPrevious value: -"Mark dimensions as primary tier. When set, the engine switches from weighted-sum blending to lexicographic ordering: the primary dimension is the sole ranking axis, and other dimensions only break ties. Use when the user says 'cheapest model, period' or similar — their stated preference becomes sacrosanct."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."benchmark5 fields changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracyAdded value: +{
+ "type": "number"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / runsAdded value: +{
+ "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+ "items": {
+ "properties": {
+ "error": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "response_text": {
+ "type": "string"
+ },
+ "test_query": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "type": "array"
+}Output schema / properties / frontier_filtered_outAdded value: +{
+ "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+ "type": "integer"
+}Output schema / properties / quality_floor_reasonAdded value: +{
+ "type": [
+ "string",
+ "null"
+ ]
+}Output schema / properties / status / descriptionAdded value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."compare2 fields changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracyAdded value: +{
+ "type": "number"
+}Output schema / properties / ab_result / properties / aggregates / items / properties / runsAdded value: +{
+ "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+ "items": {
+ "properties": {
+ "error": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "response_text": {
+ "type": "string"
+ },
+ "test_query": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "type": "array"
+}pick1 field changedOutput schema / descriptionPrevious value: -"The single best model. Same shape as rank's models[0]."New value: +"The single best model — same shape as rank's models[0]. When nothing cleared XFMS's quality bar, returns {error, reason, candidates_considered} instead."rank3 fields changedOutput schema / properties / frontier_filtered_outAdded value: +{
+ "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+ "type": "integer"
+}Output schema / properties / quality_floor_reasonAdded value: +{
+ "type": [
+ "string",
+ "null"
+ ]
+}Output schema / properties / status / descriptionAdded value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."benchmark1 field changedOutput schema / properties / models / items / properties / provider / typePrevious value: -"string"New value: +[
+ "string",
+ "null"
+]pick1 field changedOutput schema / properties / provider / typePrevious value: -"string"New value: +[
+ "string",
+ "null"
+]rank1 field changedOutput schema / properties / models / items / properties / provider / typePrevious value: -"string"New value: +[
+ "string",
+ "null"
+]comparebenchmark1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}discover1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}pick1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}rank1 field changedOutput schema / properties / xpansion_updateAdded value: +{
+ "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+ "properties": {
+ "call_count": {
+ "type": "integer"
+ },
+ "id": {
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "signup_url": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+}benchmarkdiscoverpickrank
Add one secure layer between your agents and this server.