Server Quality Checklist

Tool Scores

• call_serviceC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions calling services with examples but doesn't disclose critical behavioral traits: whether this is a mutation (likely yes, given examples), permission requirements, side effects (e.g., device state changes), error handling, or rate limits. For a tool that likely performs home automation actions, this is a significant gap in safety and operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It front-loads the core purpose and includes helpful examples in parentheses. Every word earns its place, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (4 parameters, likely mutative actions in home automation) and lack of annotations and output schema, the description is insufficient. It doesn't explain return values, error conditions, or the implications of calling services (e.g., irreversible changes). For a tool that could control physical devices, more context on safety and behavior is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 4 parameters (domain, service, entity_id, data). The description adds no additional parameter semantics beyond what the schema provides (e.g., no extra syntax, format, or usage details). Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Call') and resource ('a Home Assistant service'), with specific examples ('turn on a light, set climate'). It distinguishes from siblings like get_entity_state (which reads state) and list_entities/list_services (which list resources), but doesn't explicitly name them. The purpose is specific and actionable, though not maximally differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like get_entity_state or list_services. The description implies it's for executing actions (e.g., turning on lights), but lacks explicit when/when-not instructions or prerequisites. Usage is inferred from the examples, not stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_entity_stateC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool 'Get[s] the current state and attributes', implying a read-only operation, but doesn't specify if it requires authentication, has rate limits, or details the return format (e.g., JSON structure). This leaves significant behavioral gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is appropriately sized and front-loaded, though it could be slightly more structured by including usage hints.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and a simple single-parameter input, the description is incomplete. It doesn't explain what 'state and attributes' entail, the return format, or any error conditions, making it inadequate for full agent understanding despite the low complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, fully documenting the single 'entity_id' parameter with an example. The description adds no additional parameter semantics beyond what the schema provides, so it meets the baseline of 3 for high schema coverage without compensating value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('current state and attributes of a specific entity'), making the purpose understandable. However, it doesn't explicitly differentiate from sibling tools like 'list_entities' (which might list all entities vs. getting details for one), missing full sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as 'list_entities' or 'call_service'. It lacks explicit context, prerequisites, or exclusions, leaving the agent to infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_entitiesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states what the tool does but doesn't disclose behavioral traits such as whether this is a read-only operation (implied by 'List'), potential rate limits, authentication requirements, pagination behavior, or what the return format looks like. For a tool with no annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the purpose ('List all Home Assistant entities') and includes helpful examples ('lights, switches, sensors, etc.'). There is zero wasted text, making it appropriately sized and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete. It lacks information on behavioral aspects (e.g., safety, performance) and output format, which are crucial for an AI agent to use the tool effectively. The description covers the basic purpose but doesn't compensate for the missing structured data.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with one optional parameter 'domain' documented as filtering by domain. The description doesn't add any parameter details beyond what the schema provides (e.g., it doesn't explain the filtering further or provide additional context). With high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all Home Assistant entities') with examples of entity types ('lights, switches, sensors, etc.'). It distinguishes the tool from siblings like 'get_entity_state' (which gets state of a single entity) and 'list_services' (which lists services rather than entities). However, it doesn't explicitly differentiate from 'call_service' beyond the resource focus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for listing entities, but provides no explicit guidance on when to use this tool versus alternatives like 'get_entity_state' for single entities or 'list_services' for services. The context is clear (listing entities), but no exclusions or specific scenarios are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_servicesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the action ('List') but doesn't describe what 'list' entails—such as whether it returns a filtered subset, includes metadata, requires authentication, or has rate limits. This leaves significant gaps for a tool that presumably returns data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action without any wasted words. It's appropriately sized for a simple tool with no parameters, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is minimally adequate but incomplete. It states what the tool does but lacks details on behavior, output format, or context relative to siblings, which could hinder an agent's ability to use it effectively without additional inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so the schema fully documents the lack of inputs. The description adds no parameter information, which is appropriate here, but it doesn't compensate for any gaps since there are none. A baseline of 4 is given as the description doesn't need to cover parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all available Home Assistant services'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'list_entities' or 'get_entity_state', which would require a more specific scope comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'list_entities' or 'call_service'. It lacks context about prerequisites, timing, or any exclusions, leaving the agent to infer usage based on tool names alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.