Server Quality Checklist

Tool Scores

• detect_error_anomaliesB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the algorithmic approach (Z-score frequency analysis) but does not explain output format, how anomalies are reported, or any limitations. It also does not explicitly state this is a read-only operation. This is insufficient for a tool with no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that efficiently communicates the core purpose. Every word earns its place, with no redundant filler or repetition of schema information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 7 parameters, no output schema, and zero annotations, the description only provides a high-level overview. It does not explain what the tool returns (e.g., flagged windows, anomalies), how to interpret results, or prerequisite requirements beyond the schema. This is incomplete for a complex analysis tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description does not add meaning beyond the schema; it mentions 'time window' which maps to windowMinutes, but parameters like zScoreThreshold and lineStartPattern are not elaborated. The schema already documents all parameters clearly, so no additional value is provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Z-score frequency analysis to find sudden error spikes by time window.' It uses a specific verb (find) and resource (error spikes by time window), and distinguishes itself from sibling tools like query_log_pattern or summarize_log_timeline by focusing on anomaly detection via statistical analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for detecting error spikes but provides no explicit guidance on when to prefer this tool over siblings or when not to use it. It names the scenario (sudden spikes) but does not mention alternatives or exclusions, leaving usage inferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discover_log_schemaB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It reveals the tool analyzes a file and produces schema info, but does not disclose whether it modifies anything, what the return format is, or any limitations. This is a significant gap for a tool that reads files and infers structures.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence with no fluff or repetition. Every phrase contributes useful information about the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema and no annotations, the description should explain what the result looks like, how sample_size affects behavior, or any other operational details. It only states the high-level purpose, leaving a knowledgeable agent uncertain about the tool's output and constraints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the input schema already documents both file_path and sample_size. The description adds no additional parameter-specific meaning beyond what the schema provides, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Analyze' and the resource 'log file', with specific output details: infer format and type schemas, key type polymorphism, and regex patterns for timestamps. This distinguishes it from sibling tools like query_log_pattern or detect_error_anomalies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose implies this should be used when you need to understand the structure of a log file, but there is no explicit when-to-use guidance or mention of alternatives. Siblings exist but are not referenced, so usage context is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_external_logsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden for behavioral disclosure. It discloses that search patterns are translated and results mapped to OpenTelemetry format, which is a useful behavioral detail. However, it does not explicitly state read-only status, authentication requirements, rate limits, or error handling. The verb 'Query' implies a read operation, but this is not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no redundant words. It front-loads the main action ('Query external log providers') and concisely mentions the translation and mapping behavior. It is appropriately sized and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has four parameters, no output schema, and no annotations. The description covers the core purpose and mentions the OpenTelemetry mapping, hinting at output format. However, it lacks details on response structure, pagination, or usage constraints. The 100% schema coverage helps, but the description alone is not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides descriptions for all four parameters, achieving 100% schema description coverage. The description itself does not add parameter-specific details, but the baseline of 3 applies because the schema already handles the semantics adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'Query' with a specific resource ('external log providers') and enumerates the providers (Datadog, Splunk, Elasticsearch), making its purpose clear. It also mentions translation to OpenTelemetry format, which adds specificity. However, it does not explicitly distinguish from sibling tools like query_log_pattern, though the 'external' qualifier provides implicit differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for querying external log providers, which is a clear context. However, it does not explicitly state when to use this tool versus alternatives like query_log_pattern or summarize_log_timeline, nor does it provide exclusions. The guidance is implied but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_log_patternA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It fails to mention important behaviors such as the case-insensitive substring matching (which is only in the schema), the support for multiline stack trace reconstruction via lineStartPattern, and the exact output format or ordering. The phrase 'top N' is vague about what ordering is applied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler words. It states the core functionality in 13 words and is easy to scan. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the main purpose but omits significant context such as the lineStartPattern behavior for multiline logs and the absence of an output schema means the return format is under-specified. For a tool with 5 parameters and no output schema, more detail would be needed for full completeness, but the schema mitigates some gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema thoroughly documents all 5 parameters. The description adds no additional meaning beyond the schema; 'field/value pattern' loosely maps to the field and value parameters, but does not clarify syntax or edge cases. Baseline 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Filter' and identifies the resource 'NDJSON log file' and the action 'by field/value pattern, return top N matching entries.' This clearly distinguishes it from sibling tools like discover_log_schema or detect_error_anomalies, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a filtering use case but provides no explicit guidance on when to use this tool versus alternatives. It does not mention when not to use it or name any sibling tools, so the usage context is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• group_semantic_patternsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of disclosing behavioral traits. It reveals the algorithm and the type of output (core events and parameter distributions), but does not state whether the tool is read-only, has side effects, or requires any preconditions. This is partially transparent but leaves significant behavioral aspects undisclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the core purpose and algorithm. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description should ideally explain return values; it hints at 'core events and parameter distributions' but does not specify the structure. It also does not contextualize usage among the sibling tools. Given the moderate complexity (4 parameters), the description is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All parameters have descriptions in the input schema, providing full coverage. The tool description adds the algorithm name and the concept of parameter distributions, but does not elaborate on the individual parameters. Therefore, it meets the baseline of 3 without additional value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool clusters similar log messages using the Drain algorithm, specifying the resource (log messages) and the outcome (isolate core events and parameter distributions). This distinguishes it from sibling tools like query_log_pattern or detect_error_anomalies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a use case for grouping similar log messages but does not explicitly state when to prefer this tool over alternatives like query_log_pattern or discover_log_schema. There is no mention of exclusions or prerequisites, so the guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_log_timelineA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It communicates the core read-only aggregation behavior but omits details about output structure, adaptive sampling, or performance implications for large logs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It earns its place by immediately conveying the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 6 parameters and no output schema, yet the description only offers a one-line summary. It lacks detail on return structure and usage context, though the schema covers parameters adequately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description does not add parameter meanings beyond the schema; 'severity' and 'time window' map to levelField and windowMinutes, but these are already described in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool aggregates log severity levels into counts per time window, using 'chronological severity aggregation' as a specific verb+resource. This distinguishes it from sibling tools focused on schema discovery, pattern querying, anomaly detection, and live triage.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to choose this tool over siblings. The intended use is implied by the aggregation phrasing, but there are no alternatives or exclusions mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_live_triageA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of disclosing behavior. It mentions 'background' and 'safety limits', which hint at long-running operation and automatic shutdown, but it does not explain lifecycle details such as how the process is stopped, whether it returns immediately or streams output, or what happens when the anomaly threshold is exceeded. The schema fills some gaps via parameter descriptions, but the main description omits these behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the core action and includes important qualifiers without redundancy. Every phrase earns its place: 'background log tailing' defines the resource, 'real-time Z-score anomaly alerting' specifies the alerting mechanism, and 'heap memory safety limits' communicates a key safety feature.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema or annotations, so the description must compensate for return behavior and side effects, but it does not explain what happens after start (e.g., job ID, streaming, termination). It sufficiently covers the initiation and key features, but given the complexity of a background monitoring tool, the lack of lifecycle or return information leaves the description incomplete for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers all three parameters with clear descriptions, so the baseline is 3. The description adds no new syntax or format details beyond what the schema already provides; it merely mirrors the concepts of Z-score alerting and heap memory limits. Thus, it adds little semantic value over the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Start' with a clear resource ('background log tailing') and adds key distinguishing features: real-time Z-score anomaly alerting and heap memory safety limits. This clearly differentiates it from siblings like query_log_pattern or detect_error_anomalies, which suggest one-off or batch operations rather than a continuous background process.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies this tool is for live, ongoing monitoring rather than historical queries, giving context for when to use it. However, it does not explicitly state when not to use it or mention alternatives, such as using detect_error_anomalies for batch analysis, so it falls short of offering full usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• correlate_requestA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of transparency. It accurately describes the core behavior (collecting events, chronological sorting) but does not disclose output format, error handling, or confirm the operation is read-only. This is a moderate disclosure level.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. Every phrase adds value: the verb, the resource, the mechanism, and the sorting order.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is reasonably complete for a moderately complex tool: it explains the main action and outcome. However, since there is no output schema, it does not explicitly state the return structure, and it omits details about edge cases (e.g., no matches found). Still, the core purpose is fully covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so the baseline is 3. The description adds minimal parameter semantics beyond the schema—it only reinforces that multiple files and a trace ID are involved. It does not elaborate on optional parameters like idField or lineStartPattern, which the schema already covers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Reconstruct') and resource ('distributed trace'), and clearly distinguishes itself from siblings by focusing on collecting log events across multiple NDJSON files. It explains the method (matching trace/request ID, sorted chronologically) and is not a tautology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a clear context: use this tool when you need to reconstruct a distributed trace from multiple NDJSON files. However, it does not explicitly state when not to use it or mention alternative sibling tools, so it lacks exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.