Server Quality Checklist

Tool Scores

• control_deviceB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the action ('turning it on or off') but lacks critical details: it doesn't mention permissions required, whether this is a destructive operation (e.g., if turning off a device has irreversible effects), rate limits, error handling, or what happens upon success/failure. For a mutation tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose, and the 'Args' section efficiently documents parameters without unnecessary details. Every sentence earns its place, making it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, mutation operation, no output schema), the description is partially complete. It covers the basic purpose and parameter semantics but lacks behavioral details (e.g., side effects, permissions) and usage guidelines. Without annotations or output schema, it leaves gaps that could hinder an agent's ability to use the tool effectively in varied contexts.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful semantics beyond the input schema, which has 0% description coverage. It explains that 'entity_id' is a Home Assistant entity ID with a specific format ('domain.entity') and that 'state' accepts 'on' or 'off' values. This clarifies the purpose and constraints of both parameters, compensating well for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Control a Home Assistant entity by turning it on or off.' This specifies the verb ('control'), resource ('Home Assistant entity'), and action ('turning it on or off'), making it easy to understand. However, it doesn't explicitly differentiate from sibling tools like 'set_device_color' (which might control color instead of on/off state), so it misses the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'search_entities' (which might be for finding entities) or 'set_device_color' (which might control color settings), nor does it specify prerequisites, exclusions, or contextual cues for usage. This leaves the agent with minimal direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_device_colorA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states this is a write operation ('Set') but doesn't mention permissions, side effects, error conditions, or response format. While it specifies the action, it lacks critical behavioral details like whether this requires authentication, what happens on invalid inputs, or if changes are reversible.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by a structured parameter list. Every sentence earns its place by clarifying parameter details without redundancy. It's appropriately sized for a tool with multiple parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description covers the action and parameters well but lacks behavioral context (e.g., error handling, permissions) and return values. For a write operation with 5 parameters, this is minimally adequate but leaves gaps in understanding full tool behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It provides clear semantics for all 5 parameters: entity_id format ('light.entity'), RGB component ranges (0-255), and brightness as optional with range. This adds essential meaning beyond the bare schema, which only shows types and titles without context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Set the color and optionally brightness') and target resource ('a light entity'), distinguishing it from sibling tools like 'control_device' (more generic) and 'search_entities' (read-only). The verb 'set' is precise and indicates a write operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for controlling light color/brightness but doesn't explicitly state when to use this tool versus 'control_device' (which might handle other device types or operations) or 'search_entities' (for discovery). No explicit alternatives, prerequisites, or exclusions are provided, leaving usage context somewhat ambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_entitiesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the search operation and return format (list of entity IDs with friendly names or error message), which adds useful context. However, it lacks details on permissions, rate limits, or error conditions, leaving some behavioral aspects unspecified for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with a clear purpose statement followed by structured sections for arguments and returns. Every sentence earns its place by providing essential information without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no output schema, no annotations), the description is mostly complete. It covers purpose, usage, parameter semantics, and return values adequately. However, it could benefit from more behavioral details (e.g., search scope, limitations) to fully compensate for the lack of annotations and output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning beyond the input schema, which has 0% coverage. It explains the 'description' parameter as a natural language description with examples ('office light', 'kitchen fan'), clarifying its purpose and format. This compensates well for the schema's lack of documentation, though it doesn't cover all possible edge cases.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Search') and resource ('Home Assistant entities'), and distinguishes it from siblings by focusing on search functionality rather than control or configuration. It specifies the search is based on natural language descriptions, which is a distinct operation from the sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (searching for entities by natural language description), but does not explicitly mention when not to use it or name alternatives. It implies usage for discovery purposes, which is helpful but lacks explicit exclusions or comparisons to sibling tools like control_device or set_device_color.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.