Server Quality Checklist

Tool Scores

• send_commandB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description alone must disclose behavioral traits. It does not state that this is a mutating operation, whether commands are asynchronous, if there are side effects, or what the return value is. The examples hint at state changes but the description lacks explicit transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the main action and provides illustrative examples. It is concise and easy to scan, though not particularly rich in detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 parameters, one nested object, no output schema, and no annotations, the description covers the core purpose but omits expected details like return behavior, side effects, or failure modes. It is minimal but adequate for simple command dispatch.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so all parameters are already described in the input schema. The description adds example actions (relay.on, led.blink, set a value) that align with the schema's 'action' description but does not add deeper semantics beyond that.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Send a command to a device') with concrete examples (turn on a relay, blink an LED, set a value), which distinguishes it from sibling read/listing tools. However, it does not explicitly differentiate from push_rules or other control-adjacent tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The tool's intended use is implied by 'send a command' and the examples, but there is no explicit guidance on when to use this versus alternatives like read_device or push_rules. No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_telemetryB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for behavioral disclosure, but it reveals nothing beyond the core read action. It does not mention return format, ordering, limits, permissions, or whether readings are aggregated. The word 'Get' implies read-only but no side-effect details are confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no verbose or redundant wording. Every word contributes to the core meaning, making it perfectly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple get tool with fully described parameters, the description gives a minimal but sufficient overview. However, with no output schema, it does not explain the shape of the reading data (e.g., time series, metric grouping), which is a notable gap. The period parameter is covered by the schema enum, but return semantics remain unspecified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the parameters are well-documented in the schema itself. The description adds marginal semantic value by noting the time period, but it doesn't clarify metric filtering or the meaning of device_id beyond the schema's own descriptions. Baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get'), resource ('historical sensor readings'), and object scope ('for a device over a time period'), clearly distinguishing this from sibling tools like read_device (likely current state) and send_command. The term 'historical' adds precise semantic differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. It only states the action, leaving the agent to infer use cases. No exclusions or comparison to siblings such as 'use read_device for current state' are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_deviceB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden for behavioral disclosure. 'Get' implies a read-only operation and 'current' suggests point-in-time data, which is helpful. However, it does not reveal error behavior, authentication needs, or whether the state includes mutable settings, so some transparency gaps remain.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that directly states the verb, resource, and scope with no redundancy. It is appropriately concise and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter read tool, the description covers the core purpose, but it does not specify the return format, which is more significant given there is no output schema. It also lacks differentiation from get_telemetry, leaving some ambiguity in the tool's overall position.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema fully describes device_id with 'Device ID or partial name' and 100% coverage. The description adds no extra meaning beyond reinforcing the target is a specific device, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and the resource ('current sensor readings and state for a specific device'). It distinguishes itself from list_devices (listing) and send_command (writing), though it may overlap with get_telemetry without explicit differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus get_telemetry or send_command. The phrase 'for a specific device' implies targeting a single device, but it does not name alternatives or mention exclusions, leaving the agent to infer appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• push_rulesA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It mentions rules run on-device and trigger alerts, but does not reveal important side effects like whether pushing rules overwrites existing rules, validates syntax, requires device online, or may cause repeated alerts. This is a significant transparency gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action, and every word adds value. No redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has moderate complexity with nested rule objects. The description covers the core purpose but lacks details about rule syntax, push behavior (overwrite vs. append), validation, and error scenarios. With no output schema, the description should provide more behavioral context to be fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for top-level parameters (device_id and rules), so the baseline is 3. The description adds meaning to 'rules' by explaining they are JettyScript automation rules for sensor thresholds, but does not elaborate on the individual rule fields (id, when, then) beyond their schema names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Push JettyScript automation rules to a device' which is a specific verb+resource, and explains the purpose: rules run on-device and trigger alerts on sensor thresholds. This distinguishes it from sibling tools like send_command or get_telemetry.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for configuring on-device automation rules, as opposed to one-off commands or telemetry reads. It does not explicitly exclude use cases or compare alternatives, but the context of 'rules run on-device' and 'trigger alerts' provides adequate guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_devicesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of transparency. It accurately implies a read-only operation via 'list' and specifies the returned fields (status, last seen time), but it omits details about pagination, ordering, or any potential performance considerations. The description is not misleading, but it adds minimal context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, sharply written sentence that front-loads the action ('List') and key output details. It uses no unnecessary words and is appropriately concise for such a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter list tool, the description covers the essential return values (status, last seen time) and scope ('all IoT devices'). However, it does not describe the full device object structure (e.g., device identifier) or any limitations, which leaves minor ambiguity. Given the lack of an output schema, a bit more detail would have made it fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema is empty. As per the baseline for 0-parameter tools, the description does not need to explain parameter semantics. The description adds no parameter information, but none is required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly defines the tool's purpose: listing all IoT devices in the user's jettyd account with status and last seen time. The verb 'list' and explicit scope 'all IoT devices' effectively distinguish it from sibling tools that focus on single devices or telemetry.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: when an overview of all devices is needed. It does not explicitly mention alternatives or exclusions, but the phrase 'all devices' implies it is for broad listing rather than targeted queries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.