Server Quality Checklist

Tool Scores

• rankA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish this as a safe, read-only, idempotent operation (readOnlyHint, openWorldHint, idempotentHint, destructiveHint=false), so the bar for additional behavioral disclosure is lower. The description adds value by describing the output structure ('weights, scores, and plain-English rationale per pick') and the intent to compare, which goes beyond what annotations capture. No contradiction exists, and the lack of side-effect discussion is acceptable given the read-only nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise: two sentences, front-loaded with the primary action and followed by a clear use-case statement. Every word earns its place; there is no fluff or repetition of schema details. The structure makes it easy for an agent to quickly grasp the tool's purpose and when to invoke it.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has only 4 parameters (1 required), a rich schema, and an output schema (indicated by context). The description covers the essential context: what the tool does, what it returns, and when to use it. It does not explicitly differentiate from 'compare' or 'benchmark' siblings, but the action 'rank' and the mention of 'shortlist with weights, scores, and rationale' sufficiently convey its role. This completeness is strong but not exhaustive given the sibling ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% – every parameter (top_n, primary, purpose, capabilities) has a detailed description with examples and behavioral semantics (e.g., 'primary' switches to lexicographic ordering; 'capabilities' filters models). The main description does not add parameter-specific meaning beyond what the schema already provides, so the baseline of 3 applies. It neither compensates nor hinders, matching the schema's thoroughness.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's core function: 'Rank LLMs for a stated purpose.' It distinguishes itself from sibling tools by noting the use case is 'to see and compare alternatives, not just one answer,' which contrasts with a pick-one tool like 'pick.' This makes the purpose specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the appropriate context: 'Use when the user wants to see and compare alternatives, not just one answer.' While it doesn't name alternative tools or specify when-not-to-use, this clear use-case guidance provides enough direction for an agent to select this tool over siblings. The absence of explicit exclusions or named alternatives keeps it just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discoverA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, openWorld, and non-destructive behavior. The description adds value beyond these by revealing the output specifics: 'Returns the inferred weights and the discovery-walk trace,' and reinforces the non-ranking behavior. This provides useful additional context without contradicting annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action and key caveat, and every clause adds value. It avoids redundancy and conveys the essential facts efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a single parameter, a rich output schema, and comprehensive annotations, the description covers the tool's purpose, behavior, and usage context. It explains what the tool does, what it returns, and when to use it, leaving no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the parameter description in the schema is already detailed, explaining both the input and its purpose. The tool description adds no additional semantics for the parameter beyond what the schema states, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Show') and resource ('quality dimensions'), clearly stating the tool's function. It explicitly says 'WITHOUT ranking any models,' which differentiates it from sibling tools like rank and pick, making its unique purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear when-to-use context: 'Useful for understanding how XFMS interprets the purpose before committing to a pick.' It also states a when-not-to-use condition ('WITHOUT ranking any models'). However, it does not explicitly name alternative tools for ranking, instead implying 'pick' as a subsequent step, so it stops short of fully explicit alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pickA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint=false, so the safety profile is covered. The description adds behavioral value by disclosing the output format ('Concise output, no list') and the single-answer nature, which is not captured in the annotations. No contradictions found.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the core function then usage guidance. Every word earns its place, with no redundant filler or repetition of schema/annotation details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with rich annotations and an output schema, the description fully covers what the tool does and when to use it. The complexity is low, and the description is sufficient to guide correct invocation without requiring additional explanation of return values (handled by the output schema).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100% and the 'purpose' parameter is thoroughly described with concrete examples ('summarizing 50-page commercial leases' vs 'summarization'). The tool description adds no additional parameter-level detail, so it appropriately relies on the schema. This meets the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Return') and resource ('the single best LLM for a stated purpose'), clearly distinguishing it from siblings like 'rank' by explicitly stating 'no list'. It is concise and unambiguous about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states when to use the tool: 'Use when the user has settled on the criteria and just wants one answer.' This provides a clear context, but it does not explicitly name alternative tools (e.g., 'rank' for lists) or give when-not-to-use guidance, so it stops short of the highest bar.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• benchmarkA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial behavioral context beyond annotations: auto-expansion of test queries (5 to 10/15), parallel execution, 15+ live LLM calls, and the fact that user-specified model names are ignored. These details enrich the readOnly/openWorld/idempotent annotations without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well-organized: core action first, then mechanics and used cases, then exclusions and cost. Every sentence adds value, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers what the tool does, how it works (query generation, parallel runs), when to use (after pick/rank), when not to (user-specified models), and cost implications. An output schema exists, so detailed return-value documentation isn't required.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single `purpose` parameter is fully documented in the schema (coverage 100%) with guidance to be concrete. The description reinforces the role ('stated purpose', 'generates representative test queries') but introduces no new semantic detail beyond the schema's parameter description, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource+scope: 'Run a live A/B test against the engine's TOP 3 PICKS for a stated purpose.' It clearly differentiates from siblings by noting the engine selects candidates from the full catalog and contrasts with `compare` for user-specified models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when-to-use and when-not-to-use guidance: 'Use AFTER pick or rank when the user wants the engine's own picks stress-tested' and 'DO NOT use this when the user has already named specific candidate models — Use compare instead.' Also mentions cost relative to rank.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compareA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes beyond annotations by detailing the probe behavior: generates 5 representative queries, runs them in parallel, drops unknown IDs, refuses if fewer than 2 resolve, and caps at 3 queries for free-tier models with an explanation of why. This adds significant context beyond the read-only, idempotent annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is thorough yet front-loaded, with the core action in the first sentence and all subsequent sentences adding unique value (e.g., free-tier behavior, cost comparison, alternative tool). No sentence is redundant or off-topic.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, return values need not be described, but the description still covers edge cases (unknown IDs, refusal, free-tier cap) and operational details that make the tool fully usable. It is complete for a tool of moderate complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though schema coverage is 100%, the description adds critical semantics: `primary` only affects commentary, `purpose` is used solely to generate queries, and `model_ids` are caller-chosen and may be dropped. It also provides concrete examples, making parameter intent much clearer than schema descriptions alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Run a live A/B test between 2–5 user-specified models for a stated purpose.' It clearly defines the tool's scope and differentiates it from siblings by explicitly stating 'NO ranking step' and directing engine-chosen candidate cases to `benchmark`.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'Use this whenever the user names specific models to compare' and 'For engine-chosen candidates, use `benchmark` instead.' It also contrasts cost with `rank` and mentions when to avoid the tool, giving clear when/when-not guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.