Server Quality Checklist

Tool Scores

• impact_clockB
  3.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states that the tool requires an explicit now timestamp and never assumes current time, and that it measures from impact start rather than ticket creation. This is meaningful behavioral context beyond the schema. It does not mention side effects or error conditions, but for a calculation tool this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no wasted words. It front-loads the purpose, then adds the key behavioral constraint (time basis and explicit now requirement). Each sentence contributes necessary information, making it concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description offers the essential purpose and a critical behavioral constraint, but it does not describe the return format, potential error cases, or how severity/service influence the result. It gives enough to understand the intent but lacks detail for full autonomous use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%, covering 'now' and 'impact_start' but leaving 'service' and 'severity' undocumented. The description reinforces the meaning of impact_start and now but does not explain the roles of service and severity in determining escalation hops. It fails to compensate for the missing schema descriptions, leaving those parameters ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the core function: determining which escalation hop is active and overdue given impact start and a specified now. It also distinguishes time measurement from ticket creation, which is a specific behavioral nuance. However, it does not explicitly differentiate from sibling tools like 'escalation_path' or 'playbook', so it stops short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings. It mentions a constraint (explicit now timestamp) but does not explain when this tool is appropriate or when another might be better. There is no statement of alternatives or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• playbookA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses a key behavioral trait: when the symptom is unknown, it falls back and reports with fell_back=true. This is valuable and goes beyond simple 'returns steps.' It also implies a read-only operation. However, it does not describe the complete return structure, error handling, or any side effects, though for a read tool this is minor.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no filler. The primary purpose is front-loaded, and the fallback condition is stated concisely with a concrete outcome (fell_back=true). Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given two simple parameters and no output schema or annotations, the description provides core information: what it returns conceptually, the fallback condition, and a note about a response field. It omits the exact return payload shape (e.g., list of steps vs. free text), but for a tool this straightforward, it's nearly sufficient. A tiny bit more about the output format would make it a 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50% (only symptom has a description). The description adds meaning by explaining that a missing/unknown symptom triggers fallback behavior, linking symptom to the fallback logic. It clarifies the role of service indirectly via 'service's general steps.' Yet it does not elaborate on allowed values, format, or examples for either parameter beyond what the schema already states for symptom.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific action and resource: retrieving the first checks from a runbook for a service. The description clearly sets it apart from siblings (ownership, escalation, impact) by focusing on runbook guidance. However, 'what the runbook says to check first' is slightly vague about whether it returns a list, a single item, or a message, so it's not a perfect 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: use this tool when you need the runbook's first steps, and it handles unknown symptoms by falling back to general steps. It does not explicitly name alternatives or state when not to use it (e.g., if you need escalation path, use escalation_path). The fallback behavior is a form of guidance, but it lacks explicit comparisons to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• escalation_pathA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses a non-default behavior ('Refuses an unknown severity rather than defaulting to the quietest one') and describes the output nature (ordered list with minute timings). No other side effects are mentioned, but the tool appears read-only and the disclosure is sufficient for a simple query.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the core purpose and output, followed by the refusal behavior. Every word earns its place, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with only 2 simple parameters and no output schema, the description covers the essential return value (who, order, timing) and the key behavioral nuance (refusal on unknown severity). Nothing critical is missing given the tool's simplicity, though explicit error handling for unknown service isn't described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50% (severity has a description, service does not). The description adds significant meaning to severity: it must be a catalog-defined value and unknown ones are refused. Service is only referenced as 'this service' without additional constraints, but the schema already defines it as a string. Overall, the description compensates well for the coverage gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific query (who to wake) for a defined resource (this service at this severity) and includes output order and timing. It clearly distinguishes from siblings like who_owns (ownership) and impact_clock (timing) by focusing on the escalation path itself.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage ('Who to wake for this service at this severity') but never explicitly says when to use this tool over siblings or when not to use it. No alternatives are named, so an agent must infer from tool names rather than get explicit routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• who_ownsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explicitly states 'Returns found=false with suggestions rather than guessing at a near match,' which is a valuable behavioral trait that helps the agent set expectations. It also implies a read-only lookup (no side effects mentioned), though it does not elaborate on authentication or error conditions. Overall, it adds meaningful behavioral context beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the primary purpose. Every clause adds value: the first sentence defines the outcome; the second clarifies input flexibility and the non-guessing behavior. No redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter, no output schema), and the description covers purpose, input semantics, and a key behavior (found=false with suggestions). It does not list the exact fields returned (e.g., team name, contact info), but 'how to reach them' implies that. Given the low complexity and lack of annotations, it is reasonably complete, though a bit more detail on the return payload would make it excellent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes the parameter as 'Service name or alias' with 100% coverage, so the baseline is 3. The description adds nuance by specifying 'the name someone would actually type during an incident,' which gives the agent guidance on the flexibility of input. This extra context—emphasizing natural incident-time phrasing—goes slightly beyond the schema's dry definition, justifying a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Which team owns a service and how to reach them right now.' This specifies a verb (owns), a resource (service), and the output (ownership and contact). It distinguishes itself from sibling tools (escalation_path, playbook, impact_clock) implicitly by focusing on ownership rather than escalation steps, runbooks, or timing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context—when you need to identify who owns a service—but it does not explicitly contrast with siblings or state when not to use this tool. There is no mention of alternatives like escalation_path or playbook. The behavior note about returning found=false provides some guidance, but it is not a clear when-to-use vs. when-not-to-use directive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.