Server Quality Checklist

Tool Scores

• query_logsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses client-side filtering (status_filter, min_response_time) and parameter defaults, but does not mention pagination, error behavior, or performance implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately long with a summary and bullet-style args. It is structured but could be more concise; the first sentence effectively summarizes the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  All 7 parameters are described with examples, and the env var requirement is noted. An output schema exists, so the lack of return value description is acceptable. Missing details on error handling or rate limits but sufficient for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description provides detailed semantics for each parameter, including examples and default values (e.g., Lucene query format, ISO timestamps). Fully compensates for missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool queries OpenSearch and returns a compact log table. However, it does not differentiate from siblings like read_logs or summarize_logs, lacking specificity on when this tool is preferred.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. It mentions the OPENSEARCH_URL requirement but does not specify when not to use it or provide example contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_logsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It mentions output aggregates and env requirements but lacks details on whether the tool is read-only, error handling, or performance implications. Destructive potential is not addressed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with a clear lead sentence followed by structured argument list. It avoids fluff, though the first sentence is a fragment. The layout is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Inputs are well-covered, and the output schema exists (not shown) so return values need not be detailed. However, missing guidance on mutually exclusive parameters (path vs index), edge cases, and error states reduces completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description carries full weight. It explains each parameter (path, index, query, time_from, time_to, limit) with context like local file vs API index, Lucene query, ISO timestamps, and default limit. This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool aggregates log summaries including top endpoints, error rates, slowest requests, and unique IPs. It also specifies two modes (file or API index), making the purpose distinct from sibling tools like query_logs or get_errors.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use file mode vs API mode and mentions required env variable for API. However, it does not explicitly differentiate from siblings or state when not to use this tool (e.g., when raw logs are needed).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_errorsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must fully disclose behavior. It reveals output content (stack traces, log bodies) and mentions two modes (local file vs OpenSearch index) with an environment variable requirement. However, it omits error handling (e.g., if both path and index are provided), return structure, and potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is front-loaded with a clear purpose statement followed by structured Args list. It is not overly verbose, though the Arg descriptions could be slightly more concise. Overall efficiently conveys necessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, two data sources, output schema present), the description covers core behavior but lacks details on parameter interactions (e.g., mutual exclusivity of path and index) and edge cases. The presence of an output schema reduces the need to explain return values, but behavioral gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries full parameter explanation. It thoroughly describes all 7 parameters, including types, defaults, and contextual details (e.g., 'ISO timestamp', 'requires OPENSEARCH_URL env'). This compensates fully for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves full details for non-200 responses (errors), including stack traces, log bodies, and pod info. This distinguishes it from sibling tools like query_logs or read_logs which are general-purpose log retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide guidance on when to use this tool versus alternatives like query_logs or read_logs. It lacks explicit when-not-to-use instructions or comparisons to siblings, forcing the agent to infer usage from the name and context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_contextA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the burden. It mentions that the 'index' parameter requires the OPENSEARCH_URL env variable and gives default values for window_seconds and limit. However, it does not disclose the return format, ordering, or behavior when no logs are found, leaving some transparency gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two clear sentences establishing purpose, followed by a bullet-style argument list. Every sentence adds value, and the structure is front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 6 parameters, no schema descriptions, and no annotations, the description covers parameter semantics and a key environmental requirement (OPENSEARCH_URL). However, it omits whether path and index are mutually exclusive, does not clarify that timestamp is effectively required, and does not address error cases or output structure. An output schema exists, but the description itself is not fully complete for an agent to use without ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the tool description compensates by listing all six parameters with explanations, including the timestamp format example, the role of pod as a filter, and default values. This adds significant meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves 'full log entries around a specific timestamp' and uses the analogy 'like grep -C for logs', which immediately conveys its purpose and distinguishes it from sibling tools like get_errors or query_logs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly frames the tool for troubleshooting and provides context on when to use it (showing logs before and after an event). It does not explicitly state when not to use it or mention alternatives, but the usage context is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_logsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses key behaviors: stripping Kubernetes metadata and duplicate fields for ~98% token reduction, auto-detecting log types, and returning a compact table. It does not mention read-only nature but implies it, and lacks details on rate limits or auth, but overall transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a concise purpose statement, followed by key transformations (stripping, token reduction, auto-detection). The Args section is clearly separated and each parameter is described in a single line. No redundant or unnecessary sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 6 parameters, no annotations, and an output schema exists, the description covers the main functionality and filtering options. It does not explain return values (handled by output schema) but could mention file path assumptions or performance notes. Overall, sufficient for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. The 'Args:' section thoroughly explains all 6 parameters with details like default (limit), applicability notes (status_filter/min_response_time only for HTTP access logs), and format (ISO timestamps for time_from/time_to), adding significant value beyond the schema types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Read an OpenSearch log file (JSON or CSV) and return compact log table', specifying the action, resource, and output format. It also distinguishes from siblings by highlighting file-based reading vs likely query-based tools like query_logs, and mentions token reduction and auto-detection of log types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives (e.g., query_logs, summarize_logs). It only describes what it does without providing conditions, prerequisites, or exclusions, leaving the agent to infer from sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.