Server Quality Checklist

Tool Scores

• get_repository_infoC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must disclose behavioral context. It only states that metadata is fetched, implying a read operation, but doesn't mention whether authentication is required, how optional parameters behave, or any side effects (e.g., rate limits). This is a thin disclosure for a tool with no annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the action and enumerates the returned fields. Every word contributes value, with no redundant phrasing or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, but the description lacks essential context: it doesn't clarify the optional parameter relationship (e.g., whether both owner and repo are needed), what happens if both are null, or the exact return structure. Since there is no output schema, the description should provide more detail about the response, which it only partially does by listing fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description entirely omits the 'repo' and 'owner' parameters. It adds no meaning beyond the property names themselves, which are self-explanatory but could benefit from clarifying how they combine (e.g., 'owner/repo' vs separate). The description fails to compensate for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Fetch') and clearly identifies the resource ('repository metadata') with a list of concrete fields (description, default branch, language, topics). It distinguishes from siblings like get_readme and list_branches, though it doesn't explicitly state those exclusions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when repository metadata is needed but provides no explicit guidance on when to use this tool versus alternatives like get_readme or list_branches. There are no when/not-to-use conditions or prerequisite context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_file_contentsB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the line-range indexing and inclusiveness behavior, which is helpful. However, with no annotations available, it does not cover other important behaviors such as read-only nature, error cases, or how repository context is resolved via parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, focused sentence conveys the core functionality without any redundant words or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With six parameters, no output schema, and no annotations, the one-line description is insufficient to fully understand how to invoke the tool correctly, especially regarding repository identification and line range parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description only indirectly references line ranges without explicitly mapping to start_line/end_line or explaining path, repo, owner, and ref. It does not compensate for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action (fetch) and resource (file contents), with a specific nuance about 1-indexed inclusive line ranges. However, it does not explicitly differentiate from the sibling get_code_fragment, which may also fetch file content or ranges.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool compared to alternatives like get_code_fragment or search_code. The context is implied but not stated, and there are no exclusions or prerequisite conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_branchesB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the output includes the latest commit sha, which is a useful output trait, and the verb 'list' implies a read-only operation. However, it does not mention pagination, the effect of max_results, or any authentication requirements, so more context would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence that clearly states the tool's purpose without redundancy. It could add a sentence about parameters without becoming verbose, but as written it is efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is minimal for a tool with 3 undocumented parameters and no annotations. It does not explain how parameters interact, what the output structure looks like beyond the SHA, or how it differs from list_commits. This is insufficient for an agent to confidently invoke the tool with correct arguments.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no explanation for the three parameters (repo, owner, max_results). It does not clarify that repo/owner identify the repository or that max_results limits the number of returned branches. The description fails to compensate for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'List' and identifies the resource 'branches' while adding the detail 'with their latest commit sha'. This clearly distinguishes it from siblings like list_commits (which lists commit history) and list_directory (which lists files).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention when to prefer list_branches over list_commits or get_repository_info, nor any exclusions or context for decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_directoryB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It only states the basic listing action and default path but omits details such as whether the listing is recursive, what entry types are included, whether hidden files are shown, and the structure of the returned data. This is a significant gap for a tool with no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no extraneous words. It clearly states the action, resource, and default location in an efficient manner.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there are 4 parameters, no output schema, and no annotations, the description is incomplete. It does not explain the return value, parameter semantics, or any behavioral caveats, leaving the agent without crucial information for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has a 0% description coverage and the description does not explain the 'ref', 'owner', or 'repo' parameters. It only tangentially mentions 'path' and 'repo root', leaving the agent to guess the meaning and relationships of the other parameters. Since coverage is low, the description should compensate but does not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('list') and resource ('files and subdirectories at a path in the repo'), clearly distinguishing it from sibling tools like get_file_contents or search_code. It also mentions the default behavior (repo root), adding useful scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (when you need to list directory contents) but provides no explicit guidance on when not to use it or how it compares to alternatives like search_code or get_file_contents. There is no mention of prerequisites or context where another tool would be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_code_fragmentB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the AST-based vs regex-based vs context-window fallback behavior, which is valuable beyond what annotations (none) provide. However, it references a match_type parameter that is not present in the input schema, creating confusion. With no annotations, it also omits return format and error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the purpose, and the second sentence provides technical detail without excessive length. However, the second sentence is dense and the unexpected match_type reference muddies clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 parameters and no output schema, yet the description covers only the basic extraction mode and leaves repo/owner/ref semantics, return type, and failure behavior undefined. Standalone, an agent would struggle to invoke this correctly for remote repositories.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning for path and symbol (the Java file and the named declaration), but repo, owner, and ref are not explained. Since schema description coverage is 0%, the description must compensate, but it does so inadequately for 3 of the 5 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it extracts one named class/method/constructor body from a Java file, which is a specific verb+resource. This distinguishes it from siblings like search_code (searching), get_file_contents (whole file), and list_symbols (listing symbols).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you need a specific named declaration body, but it does not explicitly state when to use this tool over siblings like search_code or get_file_contents. It lacks alternatives or exclusions, so only the basic context is implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_commitsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It implies a read-only operation via 'list' but does not state return format, ordering, pagination, authentication requirements, or failure behavior. This is a significant gap for a tool with no annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no fluff, front-loading the main action. It earns a high score for brevity, though the brevity sacrifices detail elsewhere.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 4 optional parameters, no output schema, and no annotations, this one-sentence description is insufficient. It does not clarify the data source (repo/owner) or result limits, making it incomplete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description only hints at the 'path' parameter ('optionally scoped to a single file path') and ignores 'repo', 'owner', and 'max_results'. With 0% schema description coverage, three of four parameters remain semantically unexplained, leaving the agent to guess their meanings.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('commits'), and adds a scoping condition ('optionally scoped to a single file path') that differentiates it from sibling tools like list_branches or get_file_contents. This makes the purpose immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly indicates when to use the tool: to see recent commits, optionally filtered by a file path. This provides enough context for a simple read operation, though it does not explicitly name alternatives or exclusions, stopping short of full guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codeA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior itself. It mentions an auth requirement (GITHUB_TOKEN) and describes the return format (matching files with highlighted fragments). However, it omits details like whether repo/owner are required, whether the search is read-only, pagination behavior, or any rate limits, so it only partially covers behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and adds relevant details (highlighted fragments, auth token) without unnecessary verbiage. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 6 parameters, no annotations, and no output schema, yet the description only covers the query string and return format. It does not clarify how to scope the search (owner/repo/path), the behavior of max_results, or the meaning of extension. This leaves significant gaps for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage and 6 parameters, but the description only clarifies 'query string' and mentions 'repo' indirectly. It doesn't explain owner, path, extension, or max_results, leaving the agent to infer their meaning from the parameter names alone. This is insufficient compensation for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Search code in the repo') and identifies the primary resource ('code') and what it returns (matching files with highlighted fragments). It clearly distinguishes this from sibling tools like get_file_contents (which retrieves file contents) and list_symbols (which lists symbols).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (when you need to find code matching a query) but does not explicitly discuss alternatives or provide when-not-to-use guidance. It gives clear context for its core function but no exclusions or comparisons to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_readmeA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It mentions the output format ('as plain text') and implicitly communicates a read-only operation, but lacks details on default branch behavior, error handling, or response structure. This is basic but non-contradictory information.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It efficiently conveys the core purpose and output format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  As a simple read tool, the description is minimally viable but misses important context such as parameter semantics and behavior when ref is omitted or no README exists. With no annotations or output schema, the description could be more helpful, but it is not severely lacking.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain the meaning of the three parameters (owner, repo, ref). While the purpose helps infer repository identity, the description provides no explicit guidance on how to construct parameters, leaving significant ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb and resource: 'Fetch the repository's README as plain text.' This distinctly differentiates it from sibling tools like get_file_contents (generic file retrieval) and search_code.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for retrieving the README file, but it does not explicitly state when to use it over alternatives like get_file_contents. There are no exclusions or alternative guidance provided, leaving usage context implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_symbolsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the output type (line ranges for declared symbols) and scopes the tool to Java files, but it does not explain behavior for non-Java files, handling of repo refs, or error conditions. This is acceptable but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single well-structured sentence that front-loads the core capability and then attaches the use case. Every word contributes value, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing tool, the description covers what it returns and why to use it, but omits parameter context (especially repo/owner/ref) and any output format beyond 'line ranges.' Without annotations or an output schema, this leaves some ambiguity for invocation in a repository context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain the meaning of path, repo, owner, or ref. The only implicit hint is 'Java file,' but none of the four parameters are semantically clarified, so the agent must infer their roles from names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('symbols') while enumerating the exact symbol types and including 'line ranges,' making the tool's function precise. It also distinguishes itself from sibling tools like get_file_contents by positioning itself as a symbol-finder for get_code_fragment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'use this to find a symbol name to pass to get_code_fragment,' giving clear contextual purpose and a downstream alternative. It does not explicitly state when not to use it, but the forward reference to get_code_fragment provides sufficient usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.