Server Quality Checklist

Tool Scores

• query_log_fileB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. 'High-speed' hints at performance but doesn't disclose behaviors like file-size limits, memory constraints, error handling on malformed SQL, or whether the tool is read-only. The max_rows default via schema suggests bloat protection, but behavioral traits beyond perf aren't meaningfully disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single concise sentence that front-loads the core capability with formats listed. Efficient and zero-waste, though it could be slightly expanded with usage guidance without losing its tightness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present and 100% parameter coverage, the description needn't explain return formats. The tool is moderate complexity (SQL query execution). The description covers the what and the formats but misses operational context like load limits, safety guarantees, or typical use patterns. Acceptable but not rich.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters. The description adds marginal value: it says 'high-speed' and names supported formats, but doesn't add syntax/format details beyond the schema. Max_rows description in schema covers bloat protection. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb+resource: 'Executes high-speed DuckDB SQL queries directly against log files'. It names the formats (JSONL, CSV, Parquet) and the engine (DuckDB), distinguishing it from sibling code-inspection and patching tools. Purpose is specific and unambiguous, though it doesn't explicitly contrast with any querying sibling since none exist.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (querying log files with SQL) but provides no explicit when-not-to-use or alternative guidance. It doesn't address prerequisites like DuckDB availability or whether this is for ad-hoc analysis vs structured reporting. Adequate but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• apply_code_patchA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses meaningful behavioral traits: AST validation, Git safety gates, and automatic rollback on invalid syntax. This is genuinely useful safety information that an agent needs before invoking a mutation tool. It could add what happens on successful application (e.g., commit behavior) but covers core risk disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. The key safety behaviors (AST validation, Git gates, rollback) are front-loaded. Efficient and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema which partially relieves the description's burden, and both params are documented in the schema. For a mutation tool with no annotations, the safety behaviors are well disclosed. However, it doesn't clarify what happens on success (does it commit? create a branch?), or whether the Git safety gates could fail and how that's surfaced. A bit more operational detail would strengthen completeness for a state-changing tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters adequately. The description adds context that this is a full-content replacement ('full updated source code content') rather than a patch/diff, which is valuable distinction. However, this is baseline territory given the high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('modifies') with a clear resource ('target Python source file') and adds value by noting it validates AST syntax and establishes Git safety gates. It's clear but doesn't explicitly distinguish from sibling tools, though the verbs differ enough (query_log_file, inspect_code_ast are read operations while this is a write).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (modifying Python source files) but provides no explicit when-to-use guidance or exclusions. It doesn't mention when to prefer alternatives like inspect_code_ast or query_log_file, though the write-vs-read distinction is implied. No prerequisites beyond the two required parameters are noted.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_code_astA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of behavioral disclosure. It states the tool parses and returns structures but doesn't disclose whether this is read-only, what happens with invalid syntax, how large files are handled, or any failure modes. For a read-only analysis tool, the lack of safety-risk concern is notable, but error behavior is undisclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single focused sentence that names the action, target, and output scope with zero wasted words. It's front-loaded with the key verb 'Parses' and efficiently conveys the purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a single self-explanatory parameter, an output schema mentioned as present, and a simple one-file read operation, the description adequately covers the essentials. It could mention error-handling behavior or format limits, but given the tool's simplicity and presence of an output schema, this is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There is only one parameter (file_path) with 100% schema description coverage, so the schema fully documents it. The description adds no further param details, but with a single self-explanatory parameter and baseline 4 for 0-param-equivalent simplicity, the minimal addition is acceptable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action (parses as AST) on a specific resource (Python source file) with clear output intent (class/function/import structures). However, it doesn't explicitly distinguish from siblings like query_log_file or apply_code_patch, though the domain is so different that confusion is unlikely.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (diagnostic analysis of Python source) but provides no explicit when-to-use guidance or exclusions. It's clear this is for inspecting code structure rather than querying logs or applying patches, but no alternatives are explicitly named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.