Server Quality Checklist

Tool Scores

• assess_playerA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and excels. It explains the tool's internal behavior: mapping percentiles to tiers, picking strengths/weaknesses, attaching caveats. It also warns against inverting goalie stats, handling NAs, and not inventing stats. This fully informs the agent of the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely verbose, containing lengthy meta-instructions for the AI agent about how to present output, guardrails, and narratives. While structured, it goes far beyond concise documentation, making it harder to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of the tool (single parameter with elaborate nested object, output schema exists), the description is exceptionally complete. It covers input structure, processing logic, and output formatting, leaving no significant gaps for an agent to infer.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% with only 'card' as an untyped object. The description compensates by exhaustively documenting the expected structure for skaters and goalies, including all fields and their meanings. This is a complete and detailed definition of the parameter's semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool's purpose clearly: 'Assess one player's card: overall tier, strengths, weaknesses, deployment, trajectory, caveats, and a one-line summary.' It distinguishes itself from siblings like compare_players by focusing on a single player's assessment, though not explicitly contrasting them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides extensive guidance on when and how to use the tool, including instructions for the AI to extract fields from the card image and pass structured data. It sets guardrails and clarifies what not to do, but does not explicitly state when to avoid this tool in favor of siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_playersA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the tool is read-only (only compares card data, no modifications), reports refusal for cross-pool comparisons, handles split decisions, and flags durability concerns. Minor lack of explicit 'idempotent' or 'non-destructive' statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is thorough but verbose, with some repetition (e.g., cross-pool refusal mentioned twice). It could be tightened while retaining key details. Front-loading the output components is good, but the instructional tone takes extra space.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given complexity, the description covers all essential aspects: purpose, constraints (position pools, split logic, durability), input parameters, output structure (gaps, edge, durability, caveats, compatibility), and integration with sibling tool 'render_report'. No gaps noted.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds meaning: explains 'card_a' and 'card_b' as card JSON objects (referencing 'assess_player'), and 'focus' as optional with examples ('offense', 'defense', 'overall', role). This aids the agent in understanding parameter values beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool compares two players, producing per-component gaps, an overall edge or split, a durability flag, and caveats. It distinguishes itself from siblings like 'assess_player' (single player) and 'adjudicate_claim' by focusing on pairwise comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when the tool is appropriate: for comparing player cards within the same position pool, and refuses cross-pool comparisons. It cautions that the tool only uses card data, not external context like trades or stats, guiding when to supplement with other sources. However, it does not explicitly contrast with sibling tools like 'assess_player'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• adjudicate_claimA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavioral traits: decomposition into assertions, grading outcomes (supported/not_supported/partial/unverifiable), requirement to surface unverifiable, and scope limitations. Adds significant value beyond structured fields.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is verbose, containing procedural steps (like offering a PDF report) that could be shortened. However, it is well-structured and front-loaded with purpose and key warnings.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (2 params, nested objects, no enums, output schema present), the description covers input format, behavior, return values, and post-processing guidance. It is fully complete for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema coverage, the description richly explains both parameters: 'card' is a JSON object and 'assertions' is an array with required keys (dimension, direction, text) and lists valid dimension ids. Adds much more meaning than the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool grades a claim about a player against the card, explaining the decomposition into assertions. It distinguishes from sibling tools like assess_player and compare_players by focusing on claim adjudication.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs agents to always route claims through this tool rather than eyeballing, and clarifies scope (only card data, no trades/contracts). Lacks explicit comparison to sibling tools but provides strong context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_metricA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses behavior: thin dictionary lookup over percentile boxes, never guesses (returns found false with message), returns specific structure, and that caveat is from a unified source. It also states the tool does not reason about players.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the main purpose. Each sentence adds value, though a bit verbose in parts. Still efficient for the information provided.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one param, no annotations, but has output schema), the description is complete. It explains input, behavior, output structure (listing fields), and limitations. With output schema existing, it doesn't need to detail return values but does so briefly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% for the single parameter 'metric'. The description adds significant meaning: metric can be a schema field name or natural phrase, provides examples, and explains behavior when input is not a card metric. Fully compensates for lack of schema description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool defines a single card metric, its measure, and its top caveat. It specifies the resource (card metric) and action (define/explain), and distinguishes from siblings like assess_player by noting it does not reason about specific players.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly provides when to use (to ground narration of a metric's meaning) and when not to (as a verdict). It notes that for player-specific questions, one should use assess_player results, suggesting an alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• render_reportA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description bears full responsibility. Discloses behavior thoroughly: returns absolute file path, creates directory if missing, filename format, server-side validation of result shape, interpretive badge, and prohibition on embedding card image. This level of detail is exceptional for behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is comprehensive and logically structured, with front-loaded purpose and clear sections for usage, parameters, and examples. However, it is somewhat verbose; some sentences could be trimmed without losing meaning. Overall, every sentence earns its place given the complexity, but there is minor room for tightening.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple template kinds, strict result requirements, interactive usage), the description is highly complete. It covers all scenarios: directory creation, file naming, server-side validation, behavior for each kind, and even the interpretive badge. The presence of an output schema (noted in context signals) means return values need no further explanation, so no gap exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds extensive meaning beyond raw schema. It explains that 'kind' selects a template with specific valid values and behaviors, 'result' must be the exact engine output (not a summary), and 'title' is optional. For interpretive kind, it details the required nested structure. This adds significant semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool renders an answer into a downloadable, styled PDF report and returns the file path. It specifies the verb 'render', the resource 'answer into PDF', and the output. The description also distinguishes from sibling tools by focusing on reporting/templating, not on the analysis itself.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit instructions on when to use: after assess/compare/claim answers, always ask user if they want PDF, generate on affirmative. Details what to pass as result (exact engine object) and what not to do (never modify). Specifies template kinds and when to use interpretive. Offers clear exclusions and alternatives, fulfilling the highest standard.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.