Server Quality Checklist

Tool Scores

• get_log_detailsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It only states a read operation, but fails to mention authentication needs, error behavior (e.g., non-existent document), rate limits, or whether the operation is safe. Minimal information for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no redundant words. It is concise, but could be slightly expanded to include usage context without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema and the description only says 'full, raw JSON'. Lacks details on return format, error handling, pagination, or limits. Given the tool's simplicity, more context would be helpful for proper invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters already described in the schema. The tool description does not add any extra meaning beyond what the schema provides, so baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the full, raw JSON of a specific log event. This verb+resource combination is specific and distinguishes it from sibling tools like search_logs (which searches) and analyze_service_health (which analyzes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus its siblings, such as search_logs for searching or analyzing tools for broader insights. The description offers no context about prerequisites, alternatives, or scenarios where this tool is appropriate or not.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_service_healthB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states aggregation but does not disclose whether the tool is read-only, requires permissions, or has any side effects. The behavioral context is insufficient for an agent to understand implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no wasted words. However, the most critical information (that it determines error vs. success rates) is embedded at the end; front-loading would improve scannability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of output schema, the description should explain the return format (e.g., a ratio or table) but does not. For a tool with 3 required params and no annotations, the description is too minimal to fully guide an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (all params have descriptions), so baseline is 3. The description adds no extra meaning beyond what is already in the schema, such as clarifying the relationship between parameters or providing usage examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Aggregate') and resource ('logs'), and clearly states the output measure ('error vs. success rates'). It distinguishes itself from sibling tools like 'search_logs' (which searches) and 'get_log_details' (which retrieves details), making its unique purpose clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for health analysis but provides no explicit guidance on when to use this tool over siblings like 'search_logs' or 'scan_code_for_apis'. No when-not-to-use or alternative directions are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_code_for_apisB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states it 'scans' and 'discovers' but does not disclose what files are scanned, what constitutes discovery, whether it modifies files, or any side effects. This lack of behavioral detail is a gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 9 words with no redundancy. It is efficiently written but could optionally include more context without becoming verbose. No structural issues.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity (1 param, no output schema, no annotations), the description is minimally complete. It lacks details on output format, file scope, and behavior. For a scanning tool, more context (e.g., what happens with discovered APIs) is expected.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter projectPath, which already provides a clear description ('Absolute path to the local source code'). The tool description adds no further detail, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'scans' and identifies the resource 'local project directory' with a clear goal 'discover REST APIs'. This distinguishes it from sibling tools like search_logs or analyze_service_health.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly suggests use when discovering APIs in local code, but provides no explicit when-to-use, when-not-to-use, or alternative recommendations. The purpose alone gives enough context for basic selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_logsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must disclose behavior. It only states 'Search logs' without mentioning read-only nature, pagination, limits, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence is very concise and front-loaded. Could include more context without being verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter search tool with no output schema or annotations, the description lacks details on return format, pagination, error handling, or operational constraints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%; all parameters have clear descriptions. The tool description adds no additional meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool searches logs with keywords and time range, distinguishing it from siblings like get_log_details (more specific) and analyze_service_health (health analysis).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies usage for log searching but provides no explicit guidance on when to use vs alternatives or when not to use. Siblings are not mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_api_dashboardA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must disclose behavioral traits. It only states 'generate', which implies creation, but lacks details on idempotency, permission requirements, side effects (e.g., overwriting existing dashboards), or return value. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with verb and object. No unnecessary words. Efficiently communicates the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter tool with no output schema, the description is adequate but lacks contextual info such as whether the dashboard is created in a specific context (e.g., workspace, index pattern), or if there are prerequisites. Could be more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already explains endpoints and dashboardName well. The description adds no additional semantic value beyond the schema, earning the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb ('Generate') and resource ('a Kibana Dashboard tracking specific endpoints'). Distinct from sibling tools which focus on searching logs, getting log details, analyzing service health, and scanning code.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose implies use when needing a dashboard for specific endpoints, but no explicit when-to-use or when-not-to-use guidance. No comparison with alternatives provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.