Server Quality Checklist

Tool Scores

• get_repository_statusA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral transparency burden. It communicates a read-only intent through 'Inspect' and explicitly describes the returned JSON structure (branch, clean flag, file lists). It does not mention error behavior or explicitly state 'does not modify files,' but for a simple status-inspection tool this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only two sentences, with the primary purpose front-loaded and the return format in a compact second sentence. There is no filler, repetition, or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless, read-only status tool, the description covers what it does and what it returns, and an output schema is present to formalize the response. Missing error-condition details are a minor gap given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is no parameter schema ambiguity to resolve; the baseline 4 applies. The description appropriately omits parameter details because none exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Inspect') and resource ('current Git repository status'), with a parenthetical listing modified/staged/untracked files. It clearly states what the tool does and the status focus implicitly distinguishes it from get_git_diff, though it does not explicitly name or contrast that sibling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage context is implied: use this when you need the current repository status such as modified, staged, or untracked files. However, there is no explicit guidance on when to choose this over get_git_diff or other siblings, so the agent must infer the appropriate selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_testsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears the full burden of explaining behavior. It discloses timeout termination and return structure, which is useful, but it does not mention potential side effects of running tests, resource usage, or whether the sandbox fully isolates the process.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded with the core purpose, and uses a clear Args/Returns structure. Every sentence contributes useful information without redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with two optional parameters, the description covers behavior, parameters, and return format. The main gap is the lack of guidance about when to use this instead of get_test_output, but the presence of an output schema and clear parameter docs make it mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must explain the parameters itself. It fully does: test_target is described as a relative file or node with a concrete example, and timeout_seconds includes its default and meaning. This adds substantial meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Execute'), a well-defined resource ('automated pytest tests'), and a context ('repository sandbox'). This clearly distinguishes it from siblings like get_test_output, which retrieves results rather than running tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description tells what the tool does but gives no explicit guidance on when to choose it over alternatives such as get_test_output or apply_patch. There is no mention of when not to use it, prerequisites, or how it relates to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codeA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses matching modes, defaults, result limits, and return format, which is solid, but it does not mention regex flavor, whether the search respects ignore files, or behavior on binary files. These are non-trivial but not critical gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a one-sentence purpose, followed by a terse Args list and a clear Returns line. Every sentence earns its place; some redundancy with schema defaults exists but is minor and supports quick scanning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a five-parameter tool with no annotations, the description covers the operation, parameters, and return shape well, and an output schema is present. It is complete enough for correct invocation, though a note on regex flavor and path resolution would make it fully self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must and does compensate by explaining each parameter's purpose and default value in the Args section. It adds meaning beyond bare schema titles, though it leaves small nuances like path semantics and regex syntax details implicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Search') and resource ('repository files') and clearly distinguishes from siblings by emphasizing content search with literal or regex matching. It also explains the tool's core distinction from read_file or list_files without needing to name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes clear when to use the tool: when you need to find matching lines across repository files, optionally constrained by a path. It omits explicit 'when not to use' guidance, but the context is unambiguous enough that an agent can select it correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• apply_patchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full disclosure burden. It discloses atomicity, accepted patch formats ('diff --git' or '--- / +++'), and the JSON return contract. It does not warn about destructive side effects or conflict behavior, but the core behavior is clearly and honestly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a one-sentence summary followed by two argument explanations and a returns line. There is no redundant phrasing, and the most important facts are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter mutation tool with an output schema, the description covers the input format, optional path targeting, atomicity, and return structure. It is slightly thin on safety context such as whether changes are reversible or how conflicts are handled, but an agent has enough information to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only bare type names, so schema description coverage is 0%. The description compensates by explaining the patch parameter as unified diff content with format examples and clarifying file_path as optional, relative, and for targeting a single file. This adds real meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description leads with a specific verb and object: 'Atomically apply a unified diff patch to repository files.' This clearly differentiates it from sibling tools like get_git_diff, which generate diffs, and read_file/search_code, which are read-only. The purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied: an agent should call this when it has a unified diff to apply to repository files. However, there is no explicit when-to-use versus alternatives guidance, such as 'use get_git_diff to inspect changes instead,' so the agent must infer the decision from sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_test_outputA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description provides useful behavioral detail beyond the schema: the 'full' flag controls truncation of stdout/stderr, and the return value is a JSON string with specific fields. The verb 'retrieve' also communicates that the operation is read-only, covering the main safety question. It stops short of describing error behavior for invalid run_id, but that is a minor gap for a retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a one-sentence summary followed by clearly labeled Args and Returns sections. Every sentence carries necessary information, with defaults and return details front-loaded and no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple two-parameter tool with defaults for both parameters and an output schema already present, so the description does not need to explain return structure, yet it does. The Args section fully explains each parameter and the Returns section states the payload shape. No critical information is missing for an agent to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no descriptions (0% coverage), and the description fully compensates by explaining both parameters. It defines run_id's default behavior (empty string = most recent run) and full's effect on output truncation, adding substantial meaning beyond the bare types and defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Retrieve test execution output and logs') and a resource ('from a previous test run'). It implicitly distinguishes itself from sibling run_tests by focusing on a previous run rather than executing tests, but it never names the alternative or states that it does not trigger a run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'from a previous test run' implies the tool is for inspecting completed runs, and the parameter descriptions clarify defaults, but there is no explicit 'when to use this vs. alternatives' guidance. No exclusions or alternative tools are mentioned, leaving usage somewhat implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_fileA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description carries the behavioral transparency burden and does a solid job: it states the default 1000-line cap, the 1-indexed line behavior, and the JSON return shape including truncation status. It does not mention error handling for missing files or invalid line ranges, but for a read operation this is a minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured, with a front-loaded purpose sentence followed by a clear Args section and a Returns section. No sentence is wasted, and every line earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity, no annotations, and the presence of an output schema, the description covers everything an agent needs: purpose, parameter semantics, defaults, line limit, and return contents. It is complete enough for reliable selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description fully compensates by explaining each parameter: path is a relative repository path, start_line is 1-indexed with default 1, and end_line is 1-indexed with default None and a 1000-line limit. This adds real meaning beyond the bare schema types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'Read text content from a file inside the repository with line windowing.' It is unambiguous and distinct from siblings like list_files or search_code, though it does not explicitly name or contrast those alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use is implied by the tool's name and first sentence, and the line-windowing parameters clarify how to read a specific range of lines. However, the description does not explicitly say when to prefer this over search_code, list_files, or get_git_diff, nor does it mention any exclusion conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_git_diffA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the behavioral burden. It discloses staged-change behavior via cached, mentions determinism, and specifies the JSON return fields. However, it does not explicitly state the default scope as unstaged changes only or that untracked files are excluded.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose and uses compact Args/Returns sections. Every sentence adds meaningful information, with no filler or unnecessary repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two optional parameters and no annotations, the description covers the operation, inputs, and return shape. It is slightly incomplete because it does not clarify the default diff scope or mention edge cases like no changes or invalid paths, and it does not guide selection among siblings.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description fully documents both parameters beyond the schema. Path is explained as a relative file/directory scope, and cached is explained as switching to staged changes with a default. This fully compensates for the sparse schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description identifies a specific action ('Retrieve') and resource ('deterministic Git diff'), and clarifies optional path and cached scoping. It is clearly distinct from siblings like get_repository_status or list_files. The purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool through its name and purpose, but it does not explicitly state when to choose this over alternatives. There is no mention of get_repository_status or other sibling tools, and no exclusions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filesA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It clearly states the return type (JSON string), the top-level return fields (total_entries and a structured list), and that files include sizes. It also clarifies recursion defaults. It could mention error behavior or edge cases, but it is transparent about the core behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a concise purpose statement and then organized into compact Args and Returns sections. Every sentence adds necessary information, and there is no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple three-parameter listing tool with an output schema available, the description is complete: it covers purpose, all parameters with defaults, and the return contract. Nothing critical is missing for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the description must compensate, and it does. It explains that directory is a relative subfolder with empty string meaning root, that recursive controls traversal into child directories, and that max_depth limits recursion depth. This adds real meaning beyond the bare property names and defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb-resource pair: 'List files and directories in the repository.' This is specific and naturally distinguishes the tool from siblings like read_file (file content access) and search_code (querying code).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides solid invocation details: the directory argument is relative to the repository root, recursion defaults to true, and max_depth defaults to 10. However, it never explicitly states when to prefer this tool over alternatives or when not to use it, leaving the routing decision mostly implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.