Server Quality Checklist

Tool Scores

• list_appliancesB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden for behavioral traits. It only mentions 'browse', implying read-only, but fails to explicitly state it's non-destructive, nor disclose any permissions or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and front-loaded, but could be slightly more structured by separating the filter option explicitly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description is adequate for a simple list tool but lacks information about return format or default sorting, which would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds 'optionally filtered by category' which is already implied by the schema's category parameter; no new meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'browse' and the resource 'appliances' with an optional category filter, distinguishing it from siblings like 'find_by_capability' and 'identify_appliance' which focus on different aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for browsing appliances, optionally filtered, but provides no explicit guidance on when to use this tool versus alternatives, nor when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_operation_packageA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It details the content of the operation package but does not disclose behavioral traits such as read-only nature, authentication needs, rate limits, or side effects. It adds value by specifying what is included but lacks behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is informative and front-loaded with the core purpose. It is slightly lengthy but each sentence adds value by enumerating package components. Could be marginally more concise, but overall well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description adequately explains what is returned (components of the package). Parameter documentation is complete via schema. The context signals indicate 2 parameters with full coverage. Description adds value for a moderately complex tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (both parameters have descriptions). The description adds minimal extra meaning: it mentions optional targeting for the 'procedure' parameter and refers to 'identify_appliance' for 'ref', but these are already in the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool retrieves a robot-executable operation package for an appliance, listing specific components (procedures, control map, verification signals, etc.). It distinguishes itself from siblings by focusing on grounded knowledge for models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when needing grounded operational knowledge ('the grounded knowledge a general model hallucinates without'), but does not explicitly state when to use this tool versus alternatives like find_by_capability or identify_appliance. No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_by_capabilityA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and description does not disclose behavioral traits like permissions, pagination, or handling of no matches. For a read/search tool, minimal disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence that immediately states purpose and provides context. No superfluous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description covers purpose and usage context well. Could mention return format, but not critical for such a specific search.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema provides 100% coverage with description of 'capability'. The tool description adds examples ('heat', 'wash', 'brew') which moderately enhances understanding, but baseline is adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Find appliances by what they DO' with examples like 'heat', 'wash', 'brew'. It distinguishes from siblings by focusing on function-based search, not identification or listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'useful when an agent has a goal but not a specific model in mind', providing clear context for when to use. No explicit alternatives, but purpose implies when-not.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• identify_applianceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It notes that the tool 'returns ranked matches with confidence' and that it is a resolve/lookup operation implying read-only, but it does not explicitly state if it has side effects or destructive potential. The description adds some transparency but leaves safety assumptions unconfirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loading the purpose, then the workflow, then the output. Every sentence adds value with no redundancy. It is appropriately sized and structured for quick agent comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters, 100% schema coverage, no output schema, and sibling tools, the description covers the main use case and workflow. It explains the role in the broader toolset (first step before get_operation_package) and mentions ranked matches. However, it could elaborate on the output format or confidence interpretation, but overall it is complete enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with each parameter having a clear description. The tool description adds little beyond the schema, only reinforcing that query and panel_labels are used for identification. With high schema coverage, baseline is 3, and the description does not significantly enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool's purpose: 'Resolve which appliance an agent/robot is looking at, from observed text or panel labels, to an OPERANDI catalog object.' It specifies the verb (resolve), resource (appliance), and distinguishes from siblings by noting this is the first call in a two-step workflow with get_operation_package.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear usage directive: 'Call this first, then get_operation_package with the returned slug.' It implies when to use (as an initial identification step) and hints at the workflow. It does not explicitly state when not to use or compare with siblings like find_by_capability, but the context is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.