Server Quality Checklist

Tool Scores

• queryA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explains how to provide the argument but does not disclose what the tool returns, whether it is a read-only operation, error behavior, or side effects. It implies a query/read action but stops short of describing the tool's actual behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the key trigger condition. The example adds useful concrete detail without excessive verbosity. It could be slightly tighter, but each sentence contributes to understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool, the description covers the basic invocation and param provenance. However, since there is no output schema and no annotations, the description should ideally mention what the tool returns or any relevant behavioral outcomes. It is minimally viable but leaves some gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines 'topic' as a required string with no description, so the description must compensate. It does so effectively by explaining how to find the topic (from previous 'discover' messages) and providing a concrete example of a topic string, giving meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to query the status of a device by providing a topic. It uses a specific verb and resource ('query status' by topic), and the example makes the intent concrete. It does not explicitly differentiate from sibling tools 'discover' and 'publish', though the focus on querying status is distinct enough.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool ('When asked status of a device') and gives a concrete prerequisite: the topic must come from previous messages from the 'discover' tool. It does not explicitly state when not to use it or name alternatives, but the contextual guidance is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• publishA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It explains the core behavior (publish MQTT topic+payload) and the device-control context. However, it does not disclose potential side effects, return values, error conditions, or whether the publish is confirmed, which are relevant for an operation with no annotations or output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences that front-load the core purpose and then add a helpful usage example. Every sentence contributes meaning; there is no redundancy or unnecessary details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple publish tool with 4 parameters, no output schema, and no annotations, the description provides a clear use case and example but omits details about optional parameters, expected return behavior, and error handling. It is adequate for basic invocation but not fully complete for risk-free autonomous use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, yet the description only explains 'topic' (with a usage example) and 'payload' by name. The optional parameters 'qos' and 'retain' are not mentioned at all, and no additional meaning is added for payload. Given the low schema coverage, this is insufficient for full parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool publishes a message to a device using MQTT topic and payload. It explicitly distinguishes itself from siblings by linking usage to controlling a device and referencing the 'discover' tool for topic discovery. The example reinforces the purpose and scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: use when asked to control a device, with the topic discovered from the 'discover' tool. Includes a concrete example. However, it does not mention when not to use it (e.g., for reading data, which may be covered by 'query'), leaving a slight gap in alternative selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discoverA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the burden. It discloses MQTT topic filter usage and default scope behavior (all vs. group), but does not explicitly state read-only nature, return format, or permission requirements. 'Discover' implies non-mutating, but specifics are limited.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the core action and scope. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter discovery tool with no output schema, the description covers the primary scenario, optional behavior, and domain scope. It lacks return-shape details, but the tool is simple enough that this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds meaningful semantics for the 'group' parameter: it is a device GROUP, used when status of a group is requested, and optional with a clear default. It does not provide format examples, but compensates well for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'discover[s] devices and rules' scoped by group or entire domain, using an MQTT topic filter. This specific verb+resource+scope distinguishes it from siblings publish and query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear usage context: use a device GROUP when asking status of a group, and omit group to discover everything in the domain. It does not explicitly name alternate tools or when-not-to-use, but the guidance is practical and distinct.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.