Server Quality Checklist

Tool Scores

• get_service_metricsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only, idempotent, and non-destructive. The description adds valuable context by listing the specific metrics returned and suggesting how to interpret them (e.g., 'look for sudden spikes in error_rate or latency_ms following a deployment'), which goes beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences: the first states the exact function, the second states the use case, and the third gives an actionable interpretation tip. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has simple parameters and a clear purpose. The description covers the main use and interpretation, but it does not describe the return format or explicitly mention default time range, though the schema handles those details. Overall, it is sufficiently complete for a read-only metrics tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage for both parameters, so the schema already provides full parameter semantics. The description does not add any parameter-specific details beyond what is in the schema, making the baseline 3 appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Retrieves' and clearly identifies the resource: 'time-series performance metrics for a specific service', listing the exact metrics (latency, error rate, CPU, memory, RPS). This distinguishes it from sibling tools like get_recent_deployments or get_service_logs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using it to identify performance degradation trends and correlate with deployments or incidents, providing clear context. However, it does not mention when not to use it or name specific alternatives, so it lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_pod_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so safety is covered. The description adds behavioral depth by listing return values (desired vs ready replicas, restart count, pod phase) and providing an interpretation rule for CrashLoopBackOff with high restarts. This is valuable context beyond the annotations, though it doesn't mention failure modes or edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: the first states purpose, the second lists return fields, the third explains interpretation and usage. Every sentence contributes new information and there is no redundant or filler text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only tool with one well-documented parameter and helpful annotations, the description is complete. It covers what the tool does, what it returns, how to interpret a key signal, and how to use it during an incident alongside another tool. No output schema exists, but the description compensates by listing return fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already has 100% coverage with a clear description for the single 'service' parameter, including an example. The tool description adds no additional parameter-level guidance, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Checks the current Kubernetes pod/container status for a service.' It clearly distinguishes itself from sibling tools like get_service_metrics (metrics) and get_service_logs (logs) by focusing on pod status and replica counts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides actionable usage context: 'Use this alongside get_service_metrics to assess service health during an incident.' This names a complementary tool and a scenario. However, it does not explicitly state when not to use it or contrast with get_service_logs or get_recent_deployments, which would make it a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_recent_deploymentsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the tool as read-only, idempotent, and non-destructive, so the bar is lower. The description adds useful context by listing the exact output fields and the intended investigative use case, which helps set expectations beyond the structured annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The first sentence states functionality and filter scope, the second gives usage context and return fields. Every word contributes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (two optional params, no output schema), and the description covers what it returns, when to use it, and the filtering options. Given the annotations and schema, this is fully sufficient for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters with descriptions and defaults. The description adds only the concept of filtering by service name, which duplicates the schema, so no significant extra semantic value is provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool retrieves recent deployments across all services or by service name, with a specific verb and resource. It also distinguishes itself from sibling tools by framing itself as the first check during incident investigation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit contextual guidance: 'Use this tool first during incident investigation to check whether any recent deployment correlates with an observed symptom.' This clearly tells when to use it, though it does not explicitly list exclusions or alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• post_status_updateA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=false and idempotentHint=false. The description adds valuable behavior: the message is timestamped and attributed to 'incident-commander-mcp', and a valid incident_id is required. This goes beyond the structured annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no redundancy: it states the action, provides usage guidance, and specifies prerequisites and side effects. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter tool with annotations and no output schema, the description is complete: it covers what the tool does, when to use it, required inputs, and behavioral details (timestamping and attribution). No critical gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters (incident_id and message) having clear descriptions in the schema. The tool description reinforces usage but does not add significant new parameter-specific semantics beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb+resource: 'Appends a timeline entry to an existing incident record.' This clearly distinguishes it from sibling tools like trigger_rollback or create_incident_ticket, as it is the only tool that logs to an existing incident's timeline.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use: 'Use this to log investigation findings, actions taken, or status changes during an ongoing incident.' It implies the tool is for adding updates rather than creating new incidents or performing rollbacks, though it doesn't explicitly name alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_service_logsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the operation read-only and idempotent; the description adds the key behavioral detail that since_minutes is anchored to the simulation's current time (14:06 UTC) rather than wall-clock time. It also reveals that logs include stack traces and patterns, and there is no contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, no redundant filler. The main purpose is front-loaded, and each subsequent sentence adds either usage or parameter context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a straightforward log-read tool with no output schema, the description covers purpose, usage, time semantics, and level filtering. It doesn't describe the response format, but that's not required when no output schema exists and the use case is clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All three parameters have full schema descriptions (100% coverage), so the description adds marginal value. The notable addition is clarifying the simulation-time reference for since_minutes, which is not in the schema. Filter-by-level guidance is a usage tip rather than new semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb-resource pair ('Retrieves error logs') with service and time-window scoping. It also distinguishes itself from sibling tools like get_service_metrics or check_pod_status by focusing on log content for troubleshooting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs agents to use it after a symptom is observed and to filter for ERROR/FATAL during incidents. It implies the tool is for post-incident log inspection, but doesn't explicitly name alternative tools or exclusion cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_root_causeA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite readOnlyHint/idempotentHint/destructiveHint covering safety, the description reveals the internal cross-referencing process and defines the output structure with likelihood, confidence, and evidence, plus a confidence threshold interpretation (0.7 high certainty).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with a clear progression: what it does, how it works, when to call it. No redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool's complexity is moderate; 2 params with rich schema, and the description covers the output structure (since no output schema exists) and usage context. It explains the internal steps and confidence interpretation, making it self-sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters are fully described in the input schema with examples, and the description names them implicitly ("service", "symptom"), but adds no new syntax or format details beyond the schema. Schema coverage is 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb "Performs automated root cause correlation" and names the resource ("a service exhibiting a given symptom"). It distinguishes itself from siblings by explicitly labeling itself "the primary reasoning tool" and referencing the preceding data collection tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to "call it after collecting data with get_recent_deployments, get_service_logs, and get_service_metrics," giving a clear precondition. It positions itself as the analysis step between data gathering and action tools, though it does not list explicit when-not conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_incident_ticketA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark non-idempotent (idempotentHint=false), but the description adds valuable context: the tool appends to an in-memory registry and warns that calling twice creates two tickets. Also discloses the return value (ticket ID and full incident object). While not exhaustive about side effects, it covers the most critical behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the core action. The IMPORTANT warning is highlighted appropriately, and every sentence adds value without redundancy. It is well structured for an AI agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 simple parameters and no output schema, the description is complete: it explains the creation behavior, when to use, the duplicate risk, a pre-check resource, and the return value. This gives the agent enough context to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers all three parameters (title, severity, description) with descriptive details and validation. The description adds no additional parameter-specific meaning beyond the schema; the baseline of 3 applies since schema coverage is 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'creates' and specific resource ('new incident ticket') and explicitly mentions appending to the in-memory registry. It also distinguishes from sibling tools like 'get_recent_deployments' or 'trigger_rollback' by focusing on formal tracking creation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'when a new issue is detected that warrants formal tracking.' Also provides an exclusion by telling the agent to check for existing incidents with 'incident://current' before creating, thus preventing duplicates. This is clear contextual guidance beyond just the operation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trigger_rollbackA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the destructiveHint annotation, the description discloses critical behavioral traits: it does not immediately execute, returns a pending_approval object with a rollback_id, and requires human approval before deployment. This adds significant value not covered by annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, starting with the main purpose, followed by the important caveat and prerequisite, and ending with required fields. Every sentence adds value without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of an output schema, the description adequately explains the return value (pending_approval object with rollback_id) and the approval workflow. It also covers prerequisites and the non-execution behavior, making it complete for an agent to understand the tool's role.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the description only repeats the parameter names and example format ('service name and target_version (e.g., "v2.4.0")') without adding new semantic meaning. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifically states 'Submits a rollback request for a service to a target version,' using a clear verb and resource. It distinguishes itself from sibling tools by focusing on rollback and clarifying that it is a request rather than an immediate action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context by stating 'Call analyze_root_cause first to confirm the target version is safe to roll back to' as a prerequisite. It also sets expectations that the tool does not immediately execute, but it does not explicitly name alternative tools for immediate rollback or other scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.