Server Quality Checklist

Tool Scores

• find_failing_jobsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses the filtering behavior (only failing jobs) which is a non-trivial detail. However, it doesn't disclose what happens when no jobs fail (empty list?), whether it paginates, or whether the 'last run' reflects a specific time window. For a read-only listing tool this is modest but adequate disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, lean sentence that conveys the core purpose with zero waste. Front-loaded with the verb and resource, then the filtering criterion. Appropriate length for a focused listing tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema which relieves the need to describe return values. With one optional parameter and a clear simple purpose, the description is arguably sufficient. However, the prefix parameter is undocumented and sibling disambiguation is absent, so complete guidance would require a bit more. Overall adequate but not rich.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There is 1 parameter (prefix) with 0% schema description coverage, so the description should explain what 'prefix' filters. It does not mention the prefix parameter at all. However, the param has a default of empty string and is optional, so the agent might infer it filters job names. The description adds no meaning for the parameter beyond what the schema name implies, so this is a moderate gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description has a specific verb ('return') and resource ('scheduled jobs'), and qualifies the filter clearly ('whose last run exited with an error'). This distinguishes it from siblings like list_jobs (which likely returns all). However, it doesn't explicitly name the sibling alternative way of achieving this, so it stops short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (when you need failing jobs only), but doesn't give explicit exclusions or alternatives. Given siblings like check_recent_errors and check_log_freshness exist, some guidance on when to use find_failing_jobs vs those would help the agent disambiguate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_log_freshnessA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It explains that 26 hours for a daily job indicates non-run and frames it as a passive check. However, it doesn't disclose the return format, whether it's read-only (which is heavily implied by 'liveness check' and 'no instrumentation'), or how it handles missing directories. The behavioral traits are partially disclosed but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) and front-loaded with the purpose. The example in the second sentence is useful and earns its place. Slightly more detail on parameters or output could be added, but as written there's no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is an output schema, which reduces the burden on the description for return values. The tool is moderately complex with 3 params. The description explains the core use case and heuristic, but doesn't fully cover edge cases or clarify how it differs from tools like check_recent_errors or find_stale_placeholders beyond the cheapest-liveness framing. Adequate but with some gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description carries the burden for params. The description's 26-hour example maps to max_age_hours and implies directory is the logs location, but it doesn't explain pattern's role or clarify parameter format/edge cases. The description adds some context (the 26h heuristic) but doesn't systematically cover all three parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool reports logs that haven't been written to recently, using 'liveness check' and provides a concrete example (a daily job whose log is 26 hours old has not run). This is a specific verb+resource. It distinguishes somewhat from siblings like check_recent_errors and find_failing_jobs, though it doesn't explicitly name them as alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use it: as a liveness check for scheduled jobs. It notes this is the 'cheapest liveness check available' and 'requires no instrumentation,' implying it's a good first choice for job health monitoring. It doesn't explicitly specify when NOT to use it or name sibling alternatives, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_jobsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions this lists listings on two OSes (macOS/Linux) and returns exit status, which is useful behavioral context. However, it doesn't describe the return format details, pagination, or whether this is a pure read operation (which is likely but not stated explicitly).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise—a functional summary plus one parameter explanation with a concrete example. No wasted words, efficiently front-loaded with the core purpose. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a single optional parameter, an output schema present, and no nested objects, the tool is fairly simple. The description explains what the tool does and how the prefix parameter works, but doesn't describe the output shape or behavior on systems without launchd/systemd. For a moderately simple tool with no annotations, this is adequate but could add more behavioral detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the description must fully explain the prefix parameter. It does: 'only return jobs whose label starts with this string' with a concrete example ('com.mycompany.'). This is meaningful added value beyond the schema, which only says 'string, default empty'. The description makes the parameter's purpose and usage clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (list) and resource (scheduled jobs) and specifies it works on both launchd (macOS) and systemd (Linux), a useful cross-platform clarification. It also mentions exit status as part of the output. However, it doesn't explicitly differentiate from siblings like find_failing_jobs, though the scope is reasonably clear on its own.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage via the prefix argument example, but doesn't explicitly state when to use this tool vs alternatives like find_failing_jobs or audit. The prefix example ('skip the operating system's own jobs') gives some contextual guidance but doesn't cover exclusions or comparisons to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_stale_placeholdersA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The description does convey that this is a scanning/detection tool that searches for TODO functions with hardcoded return values, and names a detected pattern rather than just 'detecting something.' However, it does not disclose whether this could be a read-only analysis operation, whether it modifies code, the scope of scanning, or what parameters beyond directory it might depend on. No annotations make full disclosure impossible, but given the description does illuminate purpose behavior, a 3 is fair.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two paragraphs, with the first sentence stating the core purpose and the second paragraph illustrating with a concrete example. It is efficient and front-loaded. The example paragraph adds genuine value via a real detection scenario rather than fluff. No wasted sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only 1 parameter, no annotations, but has an output schema (which presumably documents results), the description covers the essential 'what does it find and why it matters' context well. It could add detail about the scope of search, common match patterns, or false positives, but for a single-parameter pattern-detection tool, the current description is reasonably complete. The concrete example fills in expected-use context that a bare schema cannot.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (the 'directory' parameter has no description), so the description must compensate. The description mentions 'reads its input file' and performs analysis in a directory context, but it never explicitly references the 'directory' parameter or explains its meaning beyond what the schema type implies (a string). The parameter is intuitive given 'directory' naming, but the description does not add semantic depth about what kind of directory this should be or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool does: 'Find functions marked TODO that still return a hardcoded value.' This uses a specific verb with a resource type (functions), and goes on to illustrate with a concrete use case (reporting function seeded with a constant). It distinguishes itself from siblings like find_failing_jobs and check_endpoint by targeting TODO-flagged placeholders specifically.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear concrete scenario ('Catches the reporting function that reads its input file, ignores it, and returns the constant'), which implies when to use it. However, it does not explicitly state when NOT to use it or name alternative tools for related but different tasks (e.g., find_failing_jobs for actual failures). The 'when to use' is implied well, but exclusions/alternatives are absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• auditA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of behavioral disclosure. It clearly indicates the tool is read-only/audit in nature ('run every check,' 'return one prioritised report'), establishing a non-destructive profile. However, it doesn't disclose behavioral details like runtime, network dependencies, permissions required, or failure behavior of individual checks, though the output schema existence helps somewhat. The core behavioral trait (aggregate, non-mutating) is conveyed, but depth beyond that is limited.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise: two short paragraphs totaling three sentences. Every line earns its place — the first states the function, the second positions it as the primary entry point, and the third gives a memorable mission framing. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex aggregate tool that runs 'every check,' the description explains the purpose well but with zero schema documentation coverage for 4 parameters. However, the output schema exists (has output schema: true), so return values don't need explaining. The main gap is parameter semantics — with 0% coverage, the description should illuminate what log_directory, code_directory, job_prefix, and max_log_age_hours control, which it does only implicitly via 'log' references. Still, the core intent and usage priority are complete enough for a first-call tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so none of the 4 parameters are documented in the schema. The description mentions log_directory implicitly (the audit inspects logs), but doesn't explain job_prefix, code_directory, or max_log_age_hours specifically. Since coverage is 0%, the description should compensate but doesn't describe what each parameter means or their default behaviors beyond what titles provide. Given the 0% coverage, this is a notable gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs every check and returns one prioritised report, with a specific verb ('run'), resource ('every check'), and outcome ('one prioritised report'). It distinguishes itself from siblings like find_failing_jobs or check_log_freshness by being the aggregate 'run everything' tool that surfaces unknown issues. The mission statement 'what on this machine is broken right now and nobody has been told?' makes the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'This is the tool to call first,' giving clear sequencing guidance. It contrasts with the granular sibling tools (find_failing_jobs, check_log_freshness, etc.) by being the comprehensive first-pass option. While it doesn't enumerate specific when-not-to-use scenarios, the 'call first' directive and the aggregate nature clearly position it against the specific checks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_recent_errorsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full disclosure burden. It transparently reveals the scoping behavior: only recent files, only last lines, and explains the rationale (avoiding stale failures). It does not reveal return format or pagination behavior, but the output schema exists to cover that gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences plus a clarifying follow-up. Every sentence earns its place: the first states the action, the second explains the scoping rationale. Zero waste, efficiently front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter tool with no annotations and 0% schema coverage, the description provides the core behavioral contract (time-boxed tail-scanning). The output schema exists, reducing the need to describe return values. It could slightly improve by explicitly defining parameter bounds, but overall it is reasonably complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate, but it only explains the 'tail' and 'recent' concepts implicitly through the prose rather than mapping to specific parameters (within_hours, tail_lines). The parameters themselves (directory, tail_lines, within_hours) are fairly self-explanatory by name and defaults, but the description doesn't explicitly define their semantics or bounds.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb+resource ('Scan the tail of recently-written logs for error signatures') and clearly distinguishes itself from siblings. The detail about 'only recent files and only their last lines' explains the behavioral scoping that differentiates it from generic log-search tools like find_failing_jobs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the when ('recently-written logs') and why the scope matters (failures fixed last week should not reappear). It implies this tool is for time-boxed error scanning, though it doesn't explicitly name sibling alternatives or state when NOT to use it. Clear context but no explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_endpointA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the full behavioral burden. It discloses a subtle non-obvious behavior: the tool inspects payload rather than status code, and that an empty/absent path denotes a failing check. It also explains the require_non_empty semantics implicitly via the empty-list example. It doesn't cover network/error behavior, but the core distinguishing behavior is clearly disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly written: two sentences of prose plus a clarifying example. The core contrast (payload not status code) is front-loaded in the first sentence. Every sentence earns its place — the require_json_path guidance and the 200-with-empty-list example are both load-bearing. Zero waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a check tool with an output schema and only one required parameter, the description is quite complete. It captures the tool's reason-to-exist (distinguishing live from disconnected integrations), explains the key toggle, and gives a concrete usage pattern. Remaining gaps (behavior when require_non_empty=false, error/handling on unexpected shapes) are minor for a validation-type tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the four parameters. It directly explains require_json_path with a concrete example ('data') and clarifies require_non_empty semantics through the empty-list scenario. However, it does not explain url, headers, or the exact meaning of require_non_empty=false, leaving some parameters under-documented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description carries a strong, distinct verb+resource: 'Check an integration by its payload, not its status code.' It clearly differentiates from sibling tools (list_jobs, check_recent_errors, etc.) by framing the check as payload-based rather than status-based. The key example (HTTP 200 with empty list = disconnected) crystallizes what makes this tool unique versus the alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives concrete guidance on how to use the tool (set require_json_path to the part proving liveness) and the motivating scenario (catching disconnected integrations behind HTTP 200). It implies when to use it versus plain status-checking tools, but doesn't explicitly name sibling alternatives or state when NOT to use it, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.