Server Quality Checklist

Tool Scores

• get_telemetryA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds significant behavioral context beyond annotations: it confirms read-only nature ('Read only'), explains pagination details (has_more, next_offset), and specifies defaults for time window and step. Annotations already indicate readOnlyHint, idempotentHint, and non-destructive, but the description enriches with concrete runtime behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: three sentences covering purpose, inputs, and output. Front-loads the primary action, then efficiently enumerates parameters and return shape. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 7 parameters and an output schema, the description is quite complete: it specifies all inputs with defaults, explains pagination, and details the return object including nested readings. Minor omission: the meaning of readings fields like 'state' could be elaborated, but output schema existence mitigates this.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with each parameter having a description in the schema. The tool description summarizes parameters but does not add substantial new meaning beyond the schema. It lists the key parameters and defaults but lacks additional semantics like edge cases or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Return time ordered sensor readings for one device across a time window.' It uses a specific verb (Return) and resource (sensor readings), and distinguishes it from sibling tools like list_devices (lists devices) and get_anomalies (returns anomalies).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives, nor does it state when not to use it. It only describes the tool's function without contextualizing its place among siblings like list_devices or get_anomalies.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_anomaliesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false. The description reinforces that the tool is read-only and adds behavioral context: 'A healthy machine returns no anomalies.' It also outlines the return structure, which is consistent with annotations. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two short paragraphs. The first sentence states the purpose, followed by input details and return structure. Every sentence adds value, and there is no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, parameters, and return structure. With an output schema present, the explanation of return values is sufficient. However, it could briefly explain how step_ms affects anomaly detection granularity, so it is not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description repeats defaults and options already present in the schema (e.g., device_id optional, step_ms default 30000, response_format). It adds minimal extra meaning beyond the schema, so a score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool detects threshold crossings for temperature or vibration over a time window. It lists inputs and output structure, and the verb 'Detect' with resource 'anomalies' is specific. It distinguishes from siblings like list_devices and get_telemetry which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool (detect anomalies) and implies that healthy machines return no anomalies. However, it does not explicitly state when not to use it or mention alternatives like get_telemetry for raw telemetry data. Still, the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_devicesA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly states 'Read only', consistent with annotations (readOnlyHint=true, idempotentHint=true, destructiveHint=false). It adds value by describing the return structure, including fields like temperature_c and vibration_mm_s.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with purpose, and includes the return structure. Every sentence adds value with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explicitly lists the return schema fields, which is sufficient even though an output schema exists. It covers all necessary aspects for a simple list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the parameter (response_format) is well-documented in the schema with enum values and default. The description does not add extra meaning beyond the schema, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List every simulated machine with its latest reading and state', providing a specific verb and resource. It distinguishes itself from sibling tools by positioning itself as the initial discovery tool for device IDs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Read only. Use this first to discover valid device ids for the other tools', providing clear guidance on when and how to use this tool relative to its siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• simulate_faultA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses key behavioral traits beyond annotations: the fault is treated as having started two minutes ago, it mutates simulator state, and returns a fault object with timing fields. This adds value to the readOnlyHint=false annotation without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with two clear paragraphs: purpose and parameter list in the first, behavioral nuance and return format in the second. Every sentence adds value, and information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (mutation with temporal effects, non-trivial return), the description provides purpose, parameters, behavioral context, and return structure. It is sufficient for an agent to correctly invoke and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with good descriptions. The description restates parameters and defaults, and adds the crucial timing behavior ('started two minutes ago') that affects how duration_seconds interacts with telemetry. This goes beyond schema, but doesn't add syntax details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool injects a fault on a device, making it visible to get_telemetry and get_anomalies. The verb 'Inject' and resource 'fault on a device' are specific, and the purpose distinguishes it from read-only sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that the tool is for making faults appear in telemetry and anomalies, and explicitly states it is not read only. While it doesn't list explicit when-not-to-use conditions, the context of siblings and the mutation warning sufficiently guide usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.