Server Quality Checklist

Tool Scores

• get_installationsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions optional pagination but does not disclose whether the tool returns all installations without filtering, what the response contains, or any other behavioral traits like default limits or ordering. This is a significant gap for a retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no redundant words or filler. It efficiently conveys the core action and the optional pagination feature. For a tool with only two simple parameters, this is appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool lacks an output schema and annotations, so the description must fully explain its use case and return value. It fails to specify that this retrieves all installations (as opposed to filtered variants), nor does it describe what the response looks like. Given the sibling tools, this omission creates confusion about when to use this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema describes both 'limit' and 'offset' parameters with defaults and meanings, providing 100% coverage. The description adds no additional parameter semantics beyond saying 'optional pagination', which is already implied by the schema. Baseline 3 is appropriate because the schema handles the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly uses the verb 'Retrieve' with the resource 'installations' and specifies the source API (SAP Cloud ALM ITSM). However, it does not explicitly state that it retrieves ALL installations or differentiate from sibling tools like get_installations_by_system_type, leaving ambiguity about its exact scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus the filtered sibling tools. It only mentions optional pagination, which is a parameter detail, not usage context. There are no exclusions, alternatives, or prerequisites mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_installations_by_customerB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must disclose behavioral traits. It only mentions the filtering action but does not explain whether the match is exact, how pagination works, or any side effects or permissions. This falls short compared to the burden placed on the description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence with no redundant words, front-loading the core purpose. It is appropriately sized for the simple functionality it describes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple filtered-list tool with robust schema coverage, the description is adequate but not fully complete. It does not explicitly mention that the result is a list of installations or clarify edge cases like empty results or exact-matching behavior, which would be useful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides clear descriptions for all parameters (customer_nbr, limit, offset), achieving 100% coverage. The description adds minimal value beyond restating that customer_nbr is used for filtering, so a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('filter') and resource ('installations') with the exact filter criterion ('customer number'). This clearly differentiates it from sibling tools like get_installations_by_system_type and get_installations_by_product.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives such as search_installations or get_installations_by_product. The intended use case is only implied by the name and description, but no direct comparison or exclusion is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_installations_by_system_typeB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only states 'Filter installations' without revealing read-only nature, pagination behavior, result format, or any potential side effects. This leaves significant behavioral ambiguity for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the core function. It contains no unnecessary words or redundant information, earning its place as a model of conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description is incomplete. It does not explain what the tool returns, how pagination works, or provide any context for when to use it relative to sibling tools. An agent would need to infer basic behavior from the name alone.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents all parameters. The description adds no extra semantic value beyond the schema, such as format specifics, constraints, or relationships between parameters. Baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Filter installations by system type', using a specific verb and resource. It also provides examples (BTP, Public Cloud) and implicitly distinguishes from sibling tools (by_customer, by_product, search) by specifying the filter dimension.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (when you need installations filtered by system type) but provides no explicit guidance or alternatives. It does not mention when to prefer this over search_installations or other filtering tools, so the guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_installations_by_productA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavioral traits. It only says 'filter', which implies a read-only operation, but does not mention pagination behavior, result size, authentication, or any side effects. This is a significant gap for a tool that could return large datasets.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that clearly states the tool's purpose without any redundant words. It is appropriately sized for a simple filter operation and front-loads the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 3 parameters and no output schema, so the description must provide enough context to fill the gaps. While the basic purpose is clear, it lacks usage guidance, return value details, and behavioral caveats. For a simple filter, this is a minimum viable description with clear gaps relative to sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes all three parameters (product_txt, limit, offset) with 100% coverage, so the description adds no additional semantic meaning. The baseline of 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'filter' and identifies the resource 'installations', with the clear scope 'by product name'. This distinguishes it from sibling tools like get_installations_by_customer and get_installations_by_system_type. It is neither a tautology nor vague.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that this tool should be used when filtering installations by product name, but it does not explicitly mention alternatives or provide when-not-to-use guidance. Since sibling tools exist for filtering by other criteria, the lack of explicit exclusions limits the agent's ability to compare options.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_installationsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. 'Search' implies a read-only operation, and the limit/offset parameters hint at pagination, but the description does not explicitly state that it's non-destructive, return format, or any rate limits. It is minimally transparent but lacks richer context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that communicates the core functionality without unnecessary detail. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is a simple search with only 3 parameters and no output schema. The description conveys the basic purpose, but given the existence of specialized sibling tools, it would be more complete if it explicitly positioned itself as the flexible alternative. Still, it is minimally sufficient for a low-complexity search.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though schema coverage is 100% (each parameter has a description), the tool description adds valuable context by listing example search fields ('system name, customer, product, etc.'), which clarifies what the query string can target beyond the generic 'Search query string'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search') and the resource ('installations'), and specifies the search scope ('by various fields' with examples). It distinguishes itself from the sibling getter tools by being the general-purpose search option, though it doesn't explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is the tool for flexible searching across multiple fields, but it does not explicitly state when to use it instead of the dedicated get_installations_by_* tools. No exclusions or alternative references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.