Server Quality Checklist

Tool Scores

• GetShardsToolC
  2.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description is the sole source of behavioral transparency. It implies a read operation ('Gets') but does not disclose whether it requires special permissions, what side effects (if any) occur, the return format, or how it handles errors or missing indices. The description adds minimal behavior beyond a bare read.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  At 10 words, the description is compact and front-loaded, with no filler. It is a single sentence that focuses attention on the core action. However, it is so brief that it sacrifices informative value, making it efficient but under-specified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema or annotation, and the description does not explain what constitutes 'information about shards' or the shape of the response. Complexity is low, but the absence of any behavioral or return details leaves the agent guessing. For a simple one-param tool, a short description might suffice, but this one is too generic to be fully contextual.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage for the 'index' parameter is 0%, and the tool description does not explain it beyond the property name. An agent can infer it likely means the index name, but the description gives no details about expected format, whether multiple indices are accepted, or how it relates to shard selection. The parameter name is self-explanatory, but the description adds no compensatory semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Gets information about shards in OpenSearch' essentially restates the tool name with minimal expansion. It does not specify what information about shards is returned (health, stats, allocation, routing), leaving the tool's exact purpose vague. It differentiates from sibling tools by resource type, but not by the specific kind of shard data provided.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool instead of ListIndexTool, IndexMappingTool, or SearchIndexTool. No prerequisites, exclusions, or alternative references are given. The agent receives no context about the appropriate use case beyond the tool's name and generic description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• SearchIndexToolB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It only mentions the action and query DSL, but does not explicitly state that the operation is read-only, mention required permissions, response format, error handling, or side effects. The term 'searches' is suggestive but not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that directly states the operation and query language without any redundant or filler content. It is well-structured and front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool's complexity is moderate due to the arbitrary nature of the DSL query, but the description provides no guidance on query construction or the return structure (no output schema). An agent would need external knowledge to correctly invoke the tool, making the description incomplete for practical use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the 'query' parameter lacks a type. The description adds that the query is in query DSL, providing partial context, but it does not explain the expected structure or format of the DSL query, nor does it detail the 'index' parameter beyond obvious inference from the tool's purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Searches' and the resource 'an index', specifying that queries are written in OpenSearch query DSL. This distinguishes it from sibling tools like ListIndexTool, IndexMappingTool, and GetShardsTool, which handle listing, mapping, and shard operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly indicates this tool is for executing search queries but does not explicitly state when to use it versus alternatives or provide any exclusion criteria. The context is clear but relies on the agent to infer distinctiveness from sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• IndexMappingToolB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. The word 'Retrieves' implies a read-only operation, but the description doesn't disclose whether permissions are needed, how errors are handled (e.g., missing index), or whether any side effects occur. There is no mention of the return format or additional behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, 12-word sentence that front-loads the purpose. Every word is useful; there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a very simple tool with one parameter, no output schema, and read-only implication. The description explains what it does and what input it takes, which covers the essential context. It doesn't mention return format or error scenarios, but the phrase 'mapping and setting information' gives a reasonable idea of the output. It could be slightly richer, but for the tool's simplicity it is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has a single 'index' parameter with no description, and coverage is 0%. However, the description's phrase 'for an index in OpenSearch' clarifies that the parameter refers to the name of the index. This adds meaning beyond the bare schema, though it doesn't provide format constraints, examples, or edge-case semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Retrieves'), the resource ('index mapping and setting information'), and the target ('an index in OpenSearch'). This distinguishes it from sibling tools like ListIndexTool (list indices), SearchIndexTool (search data), and GetShardsTool (shard info).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, exclusions, or when ListIndexTool or GetShardsTool would be more appropriate. The intended use is only implied by the tool name and the phrase 'for an index'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ListIndexToolA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility for disclosing behavior. 'Lists all indices' implies a read-only, non-destructive operation, but it does not explicitly state safety, permission requirements, pagination, or return format. The behavior is simple and apparent, yet lacks explicit disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence ('Lists all indices in OpenSearch') that is front-loaded and contains no unnecessary information. Every word contributes to the meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and no output schema, the description adequately conveys the core behavior of listing all indices. It could be more explicit about the return format or whether system indices are included, but for such a simple tool, the description is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero properties, so there are no parameters to document. The description appropriately adds no parameter information, and the baseline for zero-parameter tools is high. No further explanation is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Lists' and identifies the resource as 'all indices in OpenSearch', making the tool's purpose immediately clear. This clearly distinguishes it from sibling tools like IndexMappingTool, SearchIndexTool, and GetShardsTool, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool—whenever a list of all indices is needed—but provides no explicit guidance on when not to use it or how it compares to alternatives. There are no exclusions or alternative tool references, so the usage context is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.