Server Quality Checklist

Tool Scores

• calm_analyticsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses OData capabilities ($filter, $orderby) and an example, which is useful. But it does not mention whether the operation is read-only, return format, pagination behavior, or any side effects. This is a clear gap, so a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no redundancy. It front-loads the core function and quickly moves to a helpful example. Every sentence earns its place, and the parenthetical provider list is compact but valuable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a query tool with a rich schema (6 params, all described) and no output schema, the description is fairly complete. It gives a clear use case and mentions OData capabilities. It could arguably mention pagination or output, but the schema covers top/skip, and the example implies return of matching records. This is slightly above minimum viable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds a usage example that demonstrates filter/orderby semantics, but it mostly repeats what the schema already documents. It doesn't significantly deepen understanding of parameters beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Query') and resource ('SAP Cloud ALM analytics provider'), and lists example provider types. It also hints at distinguishing from siblings via sorted/aggregated questions, and the example 'open defects ordered by priority' makes the purpose concrete.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'use this for sorted/aggregated questions', giving a clear context. However, it does not name alternative tools like calm_list or calm_get to explicitly contrast when not to use this tool, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• calm_listA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It does disclose that OData resources accept standard query parameters while REST resources do not, and that the 'fields' parameter works for any resource by applying a projection after fetching. However, it does not mention important behavioral aspects like pagination behavior beyond OData $top/$skip, or side effects (e.g., whether listing affects system state). The description adequately covers query mechanics but not broader behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise at three short sentences. Every sentence adds unique value: the first states scope, the second distinguishes resource types, and the third gives a concrete use case. It is front-loaded with the essential information about what the tool does.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 26 parameters, 1 required, complex OData vs REST distinction, and no output schema, the description provides a good high-level overview. It covers the key architectural choice (OData vs REST) and gives a defect-specific example. However, given the absence of an output schema and the complexity of the tool, a brief mention of what the returned records look like (e.g., typical fields per resource type) would improve completeness, though the reference to calm_resources partially compensates.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, meaning all 26 parameters already have descriptions in the JSON schema. The tool description itself does not add new semantic information about parameters—it only mentions the 'fields' parameter's advantage over $select and gives a usage example for parameters like 'timebox_name'. Since the schema already describes each parameter sufficiently, the description adds some value but does not significantly enhance understanding beyond what is in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that this tool lists or queries any SAP Cloud ALM collection, naming over a dozen specific resource types. It explicitly distinguishes its scope from siblings by referencing calm_resources for the full catalog. The verb 'list or query' is specific and actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear guidance on when to use this tool by differentiating between OData resources (which accept $filter/$select etc.) and REST resources (which accept contextual params). It also gives a concrete example for listing defects (resource='tasks', task_type='CALMDEF'). However, it lacks explicit 'when not to use' guidance relative to siblings like calm_get or calm_analytics, which would improve the score further.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• calm_resourcesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears the full burden of disclosing behavior. It discloses that the tool is read-only in effect (it 'lists' information), enumerates the content categories (resources, parameters, code lists, recipes), and explains the topic parameter's effect. It does not detail output format or edge cases, but for a discovery helper this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the tool's role ('Discovery helper'), and every clause adds information about the returned content or usage. No redundant words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, no output schema), the description sufficiently covers its purpose, content, and filtering. Sibling tool names provide context that this is one of several Calm tools, and the description explains how this helper supports them.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents topic thoroughly with 100% coverage ('Optional: a resource/provider name, or "recipes" for worked examples'). The description repeats this instruction without adding new semantic detail, so it adds no value beyond the schema. Baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as a 'Discovery helper' that lists resources, parameters, code lists, and recipes accepted by sibling tools. It distinguishes itself from calm_list/get/analytics by positioning itself as a meta-tool for learning about the other tools, so the purpose is specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides direct usage instructions: pass topic='recipes' for multi-step examples or a resource/provider name to focus. This clearly states when to use the tool and how to tailor the query, though it doesn't explicitly name alternatives or exclusions beyond the implicit contrast with 'the other tools'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• calm_getA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It accurately describes the operation as a fetch (read-only) and mentions a special case for feature display ids. No destructive or side effects are implied, and the behavior is straightforward. The description is transparent about what the tool does.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. The first sentence states the purpose and a key detail (display id), the second gives the usage pattern. Front-loaded with essential information. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 3 parameters, no output schema, and no annotations, the description covers the core functionality and usage. It explains how to identify the entity (id with display id option) and which resource to use. It does not describe the output format, but for a simple fetch tool, the return (the entity object) is implicit. The description is adequately complete for an agent to select and invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (all parameters have descriptions). The description adds value beyond the schema by illustrating the usage pattern ('Choose a resource and pass its id') and providing an example of a feature display id. It also reinforces the resource enum hint by referencing calm_resources. This helps an agent understand how to compose the call correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches a single entity by id, specifies the verb 'Fetch', and gives an example for features. It distinguishes from siblings like calm_list (which lists multiple) and calm_resources (which provides resource list). The title and description align well.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage guidance: 'Choose a resource and pass its id' and references calm_resources for valid resources. It also notes that features can be fetched by display id. While it does not explicitly state when not to use the tool or name alternatives, the context is sufficient for an agent to decide when to invoke this tool versus siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.