Server Quality Checklist

Tool Scores

• set_temperatureB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It mentions 'set' implying state change but fails to describe side effects, authorization needs, failure modes, or reversibility.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with a clear one-line purpose followed by parameter explanations. No extraneous information, front-loaded with the action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with three parameters and an output schema exists, so the description need not detail return values. However, it lacks context about device prerequisites, success/failure behavior, and real-world effects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description adds value by enumerating valid values for mode ('cool', 'heat', 'auto', 'off') and unit ('F', 'C'), and explaining temperature as 'the temperature value'. This compensates for the lack of schema descriptions, though it could specify temperature range or precision.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Set thermostat temperature and mode', which clearly identifies the action and resource. It distinguishes from sibling tools like ping, get_device_status, switch_control, and lock_door, but does not explicitly differentiate usage scenarios.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus alternatives, nor does it mention prerequisites, conditions, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lock_doorA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must carry the full burden. It only states the action parameter values but does not disclose side effects, failure modes, or state behavior (e.g., what happens if already locked). Minimal behavioral info.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no redundant info. The main purpose is front-loaded, and the action values are clearly specified. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter, and the description covers its core purpose and allowed values. However, it omits return value behavior (though an output schema exists), error conditions, and any prerequisites. Adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% with a simple string parameter. Description explicitly lists allowed values 'lock' and 'unlock', adding clarity beyond the raw schema. This compensates for the lack of enum constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Lock or unlock the front door', which is a specific verb+resource pair. It distinguishes from siblings like ping, get_device_status, switch_control, and set_temperature, which are for different devices.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use lock vs unlock, or when to prefer this tool over alternatives. No prerequisites or context provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pingA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It states the tool tests server aliveness but does not disclose behavior beyond that, such as side effects, latency, or how results are returned. For a simple ping, this may be adequate but lacks explicit transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, short sentence with no unnecessary words. It is appropriately sized and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (zero parameters, no nested objects, output schema present), the description is complete enough. The output schema covers return values, so the description does not need to elaborate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so schema coverage is 100% by default. The description does not need to add parameter info, meeting the baseline of 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Test that the server is alive' clearly states the tool's verb (test) and resource (server aliveness). It distinguishes from siblings like get_device_status, switch_control, lock_door, set_temperature which are device-specific actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, such as checking device status. The description implies basic connectivity testing but offers no explicit when/when-not or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• switch_controlA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It mentions 'turn on or off' and parameter hints, but omits details like side effects, error behavior (e.g., switch not found), return format, or safety considerations. This is insufficient for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, using two short sentences that front-load the purpose and then define parameters. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is simple (two parameters, no nested objects) and an output schema exists (so return info is covered), the description provides sufficient context for typical use. A minor gap is not specifying behavior for invalid inputs, but this is partially addressed by the parameter examples.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage (no property descriptions), so the description adds essential meaning: the 'name' parameter is a partial or full name with examples, and 'action' is 'on' or 'off'. This compensates well, though it could include more details like case sensitivity or valid characters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool turns a light switch on or off, with a direct verb and resource. It distinguishes itself from sibling tools like ping (network), get_device_status (status), lock_door, and set_temperature by specifying light switches.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for light switch control but provides no explicit guidance on when to use this tool versus alternatives, nor any conditions or exclusions (e.g., prerequisites, when not to use it). The context of sibling tools helps, but the description itself lacks such guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_device_statusA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses this is a read operation ('get current status') and lists device types. However, no annotations exist, and the description does not mention potential side effects, authentication needs, rate limits, or behavior when no devices are present. It is minimally adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no extraneous information. It is perfectly concise for the tool's simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and the existence of an output schema (per context signals), the description sufficiently conveys the tool's purpose and scope. It is complete for a read-only status retrieval tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters. The description adds meaning beyond the empty schema by specifying what is retrieved (status of all listed device types), achieving baseline 4 for parameterless tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (Get) and the resource (current status of all SmartRent devices), listing specific device types (locks, thermostat, switches, sensors). It distinguishes itself from siblings like lock_door (specific device) or set_temperature (specific action).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking overall device status but does not explicitly state when to use this tool versus alternatives (e.g., for a general overview vs. controlling a specific device). No when-not or alternative guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.