Server Quality Checklist

Tool Scores

• run_diagnostic_rulesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses that rules requiring unavailable signals are skipped (not failed), explains confidence levels, and notes that findings include evidence. No side effects are mentioned but the tool likely is read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two focused paragraphs. First paragraph states purpose and key features, second provides usage guidance. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters and no output schema, the description explains the output structure (findings with evidence and confidence) and behavior (skipped rules). It covers interpretation needs well, though more detail on output fields would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 33% (only rule_ids has a description). The description adds context for rule_ids (skipped behavior) but does not elaborate on start and end parameters beyond the time range concept. Baseline is lowered due to low coverage, but the added value is marginal for the required params.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Runs'), the resource ('domain diagnostic rule set'), and the output ('structured findings, each citing the specific data samples'). This distinguishes it from sibling tools like get_signal or list_available_signals.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on interpreting low-confidence findings and checking the skipped list before concluding no problems exist. It does not explicitly compare to siblings but offers strong usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_sessionA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that it returns only structure, not interpretation, and that coverage gaps matter. No annotations are present, so the description carries the full burden. It could mention that the tool is read-only and safe to call repeatedly, but overall it's transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise, front-loaded sentences. Each sentence provides distinct value: output description, usage guidance, and a critical warning. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description covers purpose, usage, and a behavioral caveat. However, it lacks parameter details and does not hint at the output format or examples. Adequate but leaves some gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has two required parameters (start, end) with no description for how to format them or what range they cover. Schema description coverage is 0%, and the description does not add any semantic meaning for the parameters. This is a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns a structural overview of a data session, listing specific elements (signals, samples, gaps, findings by severity). It distinguishes itself from sibling tools like run_diagnostic_rules and list_available_signals.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises to use 'BEFORE detailed analysis' and directs to call run_diagnostic_rules for interpretation. Also warns about coverage_gaps being meaningful. Provides clear when-to-use and alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_available_signalsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It transparently states that signal availability depends on the data source, and that absence from the list means unavailability. It also mentions the return includes units and typical ranges. It doesn't mention side effects or auth, but for a read-only list operation, this is sufficient and clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences: first states purpose, second gives usage guidance, third sets a rule. It is front-loaded with purpose and contains no extraneous information. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately covers return format (list with units and ranges) and provides behavioral context for missing signals. The tool has no parameters, and the description is complete for its simple scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameters and schema coverage is 100%. The description adds value by confirming 'No parameters. Returns everything the current data source exposes,' and it specifies the return content (units and ranges) beyond the schema. Baseline 3 is exceeded due to extra context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'the complete list of signals available from the currently connected data source, with units and typical ranges.' It uses a specific verb ('list') and resource ('signals'), and the context distinguishes it from siblings like 'get_signal' which fetches specific signals.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises: 'Call this FIRST whenever you are unsure whether a signal exists.' It provides clear rules: if a signal is not in the list, do not attempt to retrieve, estimate, or substitute, and inform the user of unavailability and the connected source. This offers excellent when-to-use and what-to-do guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_signalA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Given no annotations, description fully discloses behavior: error conditions (unknown name, window_too_large), return format with units/timestamps, sample rate variability, decimation behavior (downsampled to max_samples, truncated flag), and warning about point reads. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Approximately 12 sentences in short paragraphs, front-loaded with purpose. No unnecessary repetition; every sentence adds value (error handling, sampling details, usage constraints). Efficient despite comprehensive information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema and 4 parameters, description covers return format, errors, sampling behavior, and constraints. No gaps remain for agent to infer dangerously; complete enough for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (all parameters have descriptions), but description adds significant context: name must be exact and case-sensitive from list_available_signals, max_samples explanation (downsampling cap, trade-offs), start/end are inclusive ISO 8601. Adds rationale and usage tips beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states verb 'retrieves', resource 'one signal over a time range', and return format 'timeseries with explicit units and timestamps'. Distinguishes from sibling tools (list_available_signals, run_diagnostic_rules, summarize_session) by focusing on signal data retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidelines: name must exactly match list_available_signals, unknown name returns error, sample rate differences, point reads (actual_sample_rate_hz null) should not be used for timing, downsampling behavior, time range bounding to avoid errors. Clearly tells when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.