Server Quality Checklist

Tool Scores

• submit_reviewA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark this as a non-read-only, non-idempotent operation. The description adds the requirement that write access must be explicitly enabled, which is a useful behavioral constraint, but it does not clarify consequences of repeated submissions or the meaning of the permission requirement. It is minimal but not contradictory.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the primary purpose, and contains no redundant words. Every clause adds information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has five parameters, no output schema, and sparse annotations, the description leaves out key details: what the response will be, whether a body is required for certain events, and how this relates to sibling tools like request_changes. It provides a basic purpose and permission note but is not fully self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 80% and the schema already describes each parameter. The description repeats the event enum values (APPROVE, REQUEST_CHANGES, COMMENT), which adds little beyond the schema's enum. It does not explain when the optional body parameter is needed or the impact of the pull_number.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (submit) and the resource (pull request review), and enumerates the three possible event types (APPROVE, REQUEST_CHANGES, COMMENT). This distinguishes it from sibling tools like request_changes or post_review_comment, which target narrower actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description only mentions a permission prerequisite ('Write access must be explicitly enabled') but does not explain when to use this tool versus alternatives like request_changes or post_review_comment. No exclusions or alternative scenarios are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_prA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, and the description adds what data is returned (merge state, labels, SHAs) but does not disclose any additional behavioral traits like error handling, rate limits, or authentication requirements. It adds some value but not rich context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that leads with the verb, enumerates output categories, and avoids redundancy or excessive detail. Every word contributes value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple, read-only fetch tool with three fully described parameters and no output schema, the description sufficiently indicates what will be returned. It could be slightly more explicit about single-PR scope, but the sibling list makes this clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with clear descriptions for owner, repo, and pull_number. The tool description does not add extra parameter-level meaning beyond what the schema already provides, so baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Fetch' with the resource 'pull request' and enumerates distinct contents (metadata, merge state, counts, labels, branch SHAs), clearly differentiating it from siblings like list_prs or get_pr_diff.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool (fetching a specific PR's data) but provides no explicit alternatives, exclusions, or context about when to prefer sibling tools. The guidance is implied by the tool name and sibling list rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_labelsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide idempotentHint=true and destructiveHint=false, but the description adds a specific non-destructive behavior: 'Existing labels are preserved.' This clarifies that labels are added, not replaced, a key behavioral detail beyond the generic annotations. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action, and every sentence earns its place. The second sentence conveys an important behavioral nuance without waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple mutation tool with annotations and a clear description, the information is nearly complete. It lacks details about response format or error handling, but given the straightforward nature and the presence of annotations, it is adequately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters with meaningful descriptions (owner, repo, pull_number, labels). The description does not add parameter-level semantics beyond what the schema already provides, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Add one or more labels to a pull request') with a specific verb and resource. It distinguishes from sibling tools (which read PRs, comment, review) by focusing on label modification, and adds a behavior note ('Existing labels are preserved') that further clarifies scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when labels need to be added to a PR, but gives no explicit guidance on when not to use it or alternatives. There are no exclusions or comparisons with sibling tools, so usage context is merely implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• post_review_commentA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate a write operation (readOnlyHint=false). The description adds the operational context that write access must be explicitly enabled, which is beyond the annotations and useful for the agent. It does not elaborate on side effects or return behavior, but with annotations present this is a reasonable level.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first states the core purpose, the second adds a necessary prerequisite. Every word earns its place, and important information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 10 parameters and no output schema, the description covers the essential action and the write-access requirement. However, it would benefit from mentioning when to prefer this over submit_review for multi-comment scenarios. The schema carries the rest of the parameter detail, so overall it is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all 10 parameters are already described in the input schema. The description adds no parameter-specific information beyond what the schema provides, so it meets the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Post a single inline comment at a diff line.' This clearly states the action, scope, and distinguishes it from siblings like list_pr_comments or submit_review.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a prerequisite ('Write access must be explicitly enabled') but does not explicitly state when to use this tool versus alternatives like submit_review. It implies the niche (single inline comment) but lacks comparative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• request_changesA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate a non-read-only, non-idempotent operation. The description adds the prerequisite about write access and emphasizes the required explanation, providing context beyond the schema. It does not contradict annotations and offers useful operational detail, though it does not describe broader side effects on the PR.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the core action. The second sentence adds a necessary prerequisite without redundancy. Every word contributes value, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, 4 required parameters, no output schema, and existing annotations, the description covers the essential purpose and prerequisite. It does not elaborate on effects or return values, but these are not critical for this straightforward action. A brief note distinguishing from submit_review would have made it more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage for all four parameters. The description adds no new parameter-level details beyond what the schema already states (e.g., body is required explanation). With full schema coverage, a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Submit a REQUEST_CHANGES review' with a specific verb and resource. It distinguishes this from sibling tools like post_review_comment or submit_review by naming the exact review type. The title reinforces the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers a prerequisite: 'Write access must be explicitly enabled.' However, it does not explicitly state when to prefer this tool over alternatives like submit_review, though the name and description imply specialized use for requesting changes. No exclusions or alternative guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_pr_diffA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=true, so the read-only nature is covered. The description adds valuable behavioral context by explaining pagination via offset/limit and noting that the diff is 'raw', which goes beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with the core action front-loaded. Every word contributes to understanding the tool's function and usage. No redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only tool with no output schema, the description specifies the output ('raw unified diff') and covers the main usage nuance (pagination). It is complete enough for the agent to select and use the tool correctly, though it could mention defaults or return format more explicitly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents all parameters. The description adds meaning to offset and limit by explicitly describing their role in paging through large pull requests, which is more specific than the schema's basic descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Fetch' and resource 'raw unified diff', clearly distinguishing it from sibling tools like get_pr (which would retrieve pull request metadata). The purpose is immediately obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides guidance on using offset and limit for large pull requests, but it does not explicitly state when to use this tool versus alternatives such as get_pr or other sibling tools. The usage context is implied but lacks formal exclusions or alternative recommendations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_pr_commentsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=true, covering the safety profile. The description adds meaningful behavioral context beyond the annotations: it explicitly states that both inline review comments and top-level PR conversation comments are included, and that pagination is supported. This adds value without contradicting the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that immediately states what the tool does and its key scope. Every part is informative, with no filler or redundant repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only list tool with good schema coverage and annotations, the description sufficiently conveys the tool's purpose, scope, and pagination behavior. It does not detail output fields, but the absence of an output schema and the straightforward 'list comments' pattern make this acceptable for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: all six parameters (owner, repo, pull_number, page, per_page, max_pages) have descriptions in the schema. The description does not add additional parameter semantics, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and clarifies the resource (pull request comments) and scope (both inline review comments and top-level PR conversation comments), which distinguishes it from sibling tools like list_prs and get_pr_diff.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly identifies what the tool is for (listing both comment types with pagination), giving clear context for when to use it. However, it does not explicitly mention when not to use it or name alternatives like get_pr or post_review_comment, so it misses the full 'when-not/alternatives' guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_prsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds behavioral context beyond the readOnlyHint/openWorldHint annotations by specifying 'multi-page pagination' and 'compact triage-oriented response', which informs the agent about the tool's scope and output style. This is useful for setting expectations about fetching many items and the response format. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the core function, and includes two high-value behavioral hints without fluff. It earns a 5 for structure and efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the rich schema with 100% parameter coverage and annotations indicating a safe read operation, the description covers the essential use case. The phrase 'compact triage-oriented response' gives a general sense of output, but without an output schema, a bit more detail on response fields could be helpful. Still, it is adequately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 8 parameters have schema descriptions, so the schema carries the full semantic weight. The description mentions pagination behavior but not parameter specifics, so it adds no additional meaning to individual parameters. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as 'List pull requests' with a specific verb+resource. It distinguishes from sibling get_pr by focusing on listing multiple PRs and mentions pagination and response style, which sets it apart from other PR-related tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a use case for triage-oriented overviews (compact response) and multi-page pagination, which helps an agent know when to use this tool over get_pr (which fetches a single PR). However, it does not explicitly state when not to use it or name alternatives, so it misses the full exclusion criteria for a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.