Server Quality Checklist

Tool Scores

• trigger_speedtestA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits on its own. It explains the two test types, but gives no information about side effects, whether the operation is asynchronous, expected duration, or any necessary prerequisites. This is a gap for a trigger action that likely has operational impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the purpose sentence. The parameter explanations are organized and free of any unnecessary text. Every sentence contributes to understanding how to use the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With only two parameters, both explained, the description covers the essentials. However, it omits operational context such as whether the speed test runs in the background, how results are returned (though an output schema exists), or any prerequisites like device connectivity. This is adequate but leaves some gaps for a trigger tool with no annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides no descriptions, but the description explains both parameters well: device_id is 'The Orb device ID to run the speed test on,' and test_type clarifies the meaning of 'content' and 'top'. This adds meaningful value over the bare schema and fully compensates for the 0% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb and resource: 'Trigger a speed test on an Orb Cloud device.' This is distinct from sibling tools like list_devices or get_device_telemetry, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the use case: when you need to run a speed test on an Orb Cloud device. It does not explicitly mention alternatives, but no sibling tool performs this function, so the context is clear. There are no exclusions or prerequisites stated, but the purpose is sufficient for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_organizationsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The description uses 'List,' which implies a read-only operation, and references the API key scope, but it does not explicitly state whether the operation is non-mutating, mention rate limits, or address potential pagination. The information is adequate but not rich; hence a 3.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no unnecessary words. It states the action and resource directly and effectively, achieving maximum clarity in minimal space.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is very simple: zero parameters, an output schema exists, and the description covers the purpose and scope. It doesn't mention the return format, but the presence of an output schema compensates for that. A 4 reflects that the description is largely complete for a basic list tool, though it could explicitly note that it returns a collection of organizations or any potential limitations such as pagination.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, and the schema description coverage is 100% by default. With 0 params, the baseline is 4 per the rubric. The description adds no parameter details because there are none to explain, but it appropriately mentions the API key as the implicit scope parameter, which is useful contextual meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all Orb Cloud organizations accessible with the configured API key.' The verb 'List' is specific, the resource 'Orb Cloud organizations' is precise, and the scope ('accessible with the configured API key') adds clarity. It naturally distinguishes itself from sibling tools like list_devices and trigger_speedtest, which address different domains.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context that the tool lists organizations available via the configured API key, implying it should be used for organization enumeration. It doesn't explicitly name alternatives or exclusions, but sibling tools are clearly unrelated (devices, telemetry, speedtests, datasets), making the usage intent obvious. This earns a 4 for clear context without explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• configure_temp_datasetsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and it discloses the temporary nature, automatic reversion to default after expiration, and the conditional behavior when datasets_config is omitted. It does not detail permissions or immediate activation mechanics, but the key behavioral traits are covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized with a purpose sentence, a short behavioral explanation, and an Args list. Every sentence adds value and the structure makes it easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the rich nested schema and presence of an output schema, the description need not repeat every subfield. It explains the temporary lifecycle and parameter semantics comprehensively; some gaps remain around operational prerequisites, but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The Args section explains all three top-level parameters, including the device identifier, duration format with examples, and the role of datasets_config plus the effect of omitting it. This compensates for the low schema description coverage (33%) at the top level.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Configure') and resource ('temporary dataset collection and push for an Orb Cloud device'), and the scope is unambiguous. It clearly differs from siblings like list_devices and trigger_speedtest by describing a configuration action with temporary behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context: the tool enables temporary push to a custom endpoint or local API for a fixed duration, and notes that omitting datasets_config only updates the duration. It does not explicitly name alternatives or exclusions, but the sibling tools are clearly different in purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_devicesA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the nature of the data ('data that changes rarely') and the return categories (identity, hardware info, location, configuration), but does not explicitly mention that the operation is read-only or cover edge cases like pagination, permissions, or empty results. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a purpose sentence, a return-value clarification, and an explicit alternative. The Args block is cleanly separated. Every sentence earns its place without redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential aspects: purpose, alternative usage, parameter meaning, and output content. An output schema exists, so return details need not be fully spelled out. The only gap is missing info about pagination or result limits, which is often relevant for a list operation, but the tool is simple enough that this does not undermine completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It provides an Args section that explains 'organization_id: The organization ID to list devices for,' adding meaningful semantic context beyond the schema's bare string type. For a single parameter, this is sufficient, though it could include format hints or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'List devices in an Orb Cloud organization with stable configuration details.' It specifies what the tool does and the scope of data (stable configuration), distinguishing it from sibling tools like get_device_telemetry.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly names an alternative: 'For real-time connectivity and performance scores, use get_device_telemetry instead.' This tells the agent when not to use this tool and directs to the correct sibling, satisfying the highest bar for usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_device_telemetryA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It clearly states the operation returns real-time connectivity status and performance scores, and explains the filtering behavior with device_id. It implicitly signals a read-only operation via 'Get', though it does not explicitly mention lack of side effects or error conditions, which is acceptable for a simple read tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the purpose. The additional paragraph and Args section are efficient and add necessary details without fluff. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two parameters and an existing output schema, the description fully covers the essential behaviors: what it returns, the filtering capability, and the organizational context. The description is complete enough for an agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the tool description provides an 'Args' section that explains both parameters: organization_id as the organization to query, and device_id as an optional filter. This adds meaningful semantics beyond the raw schema, fully compensating for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool gets live telemetry for devices in an Orb Cloud organization, with the specific verb 'Get' and resource 'device telemetry'. It is distinct from sibling tools like list_devices and trigger_speedtest, and adds specifics about returning connectivity status and Orb performance scores.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates when to use (when needing live telemetry) and provides conditional usage for the optional device_id parameter. It does not explicitly reference alternatives or exclusions, but the context is clear enough for an agent to select the tool appropriately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.