Server Quality Checklist

Tool Scores

• get_data_metricsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=true, indicating safe reads. The description adds value by explaining that data is simulated by default and the need to switch to a real backend in production. However, it mentions a 'simulated' parameter not present in the input schema, which could cause confusion.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two concise sentences, front-loading the purpose and then adding behavioral notes. No unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema (not shown), the description need not detail return values. It covers purpose, funnel step, simulation behavior, and production considerations. The only gap is the mention of a 'simulated' parameter not in the schema, which slightly reduces completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage with descriptions for both parameters. The tool description does not add significant semantic meaning beyond what the schema provides. Baseline is 3 per guidelines.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: fetch current values, rolling averages, and thresholds for given metrics at a location. It also identifies itself as 'Step 3 of the funnel', distinguishing it from sibling tools like find_metrics (likely step 2) and analyse/diagnostic_categorize (post-processing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context on when to use this tool ('Step 3 of the funnel'), implying it follows find_metrics. It also mentions the simulation default and production swap, guiding usage scenarios. However, it does not explicitly state when not to use it or list direct alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diagnostic_categorizeA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, consistent with a classification read operation. The description adds behavioral detail: returns a confidence score and scores for every category, beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is front-loaded with key information ('Step 1 of the funnel') and includes all necessary details without any wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (1 parameter, output schema exists), the description adequately covers what the tool does, including the output (confidence score and per-category scores). It is complete for its purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter 'symptom' with a clear description. The tool's description does not add extra parameter meaning beyond what the schema already provides, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (classify), the input (free-text symptom), and the output (diagnostic category with confidence and per-category scores). It positions itself as 'Step 1 of the funnel,' distinguishing it from sibling tools like analyse or find_metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Step 1 of the funnel,' indicating when to use it (as a first step). However, it does not provide when-not-to-use guidance or mention alternatives, though the sibling tools imply a pipeline.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyseA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide readOnlyHint: true, and the description's actions ('evaluate', 'surface anomalies') are consistent with a read-only analysis. The description adds behavioral details like returning recommendations and next steps, which goes beyond the annotation alone.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the funnel step and clearly conveys all necessary information without any extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, the description adequately explains the tool's role, inputs, and outputs. It provides step context and mentions the return of recommendations and next steps, making it effectively complete for an analysis tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds context by noting that 'data_metrics' is the output of 'get_data_metrics'. This helps the agent understand the expected input format beyond the schema's generic description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs ('evaluate', 'surface', 'return') and identifies the resource ('metric data'). It explicitly labels itself as 'Step 4 of the funnel', distinguishing it clearly from siblings like 'diagnostic_categorize', 'find_metrics', and 'get_data_metrics'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description positions the tool as part of a sequential funnel ('Step 4'), implying it should be used after prior steps. However, it does not explicitly state when not to use it or what alternatives exist beyond the sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_metricsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint=true, and the description aligns by stating the tool returns metrics. It adds value by detailing output content (definitions, relevance), but does not discuss potential behavioral traits like rate limits or permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the funnel context and directly states the tool's action, with no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (2 simple parameters), full schema coverage, and an output schema, the description sufficiently covers the tool's purpose and expected behavior without gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with clear definitions for symptom and category. The description adds context about the tool's purpose and output, but does not elaborate on parameter semantics beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: given a symptom and category, return relevant cURL metrics with definitions and reasons. It positions itself as 'Step 2 of the funnel', distinguishing it from siblings like 'diagnostic_categorize' and 'get_data_metrics'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by referencing 'Step 2 of the funnel' and requiring symptom and category inputs, but it does not explicitly state when not to use it or compare with siblings beyond context clues.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.