Server Quality Checklist

Tool Scores

• checkB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It mentions tracking imports and returning analysis, but it does not discuss side effects, permissions, rate limits, or any constraints. The description is minimal beyond the basic purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured with a clear purpose followed by an Args block that lists each parameter. It avoids unnecessary fluff, though the Japanese text could be slightly shorter. The front-loading of the core purpose is good.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's functionality and parameter meanings adequately. However, it lacks details about the return format beyond 'detailed analysis and judgment'. Given no output schema and no annotations, more completeness would be beneficial, especially regarding expected output structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description adds meaningful context for all four parameters: entry_point (project entry file), package_name (package to investigate), python_path (site-packages path, auto-detect if empty), and max_depth (max search depth, default 5). This significantly aids the agent in understanding parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool analyzes how a specific package is used in a project, tracking import chains, gateway functions, and call status. It uses a specific verb ('analyze') and resource ('package'), and while the sibling 'analyze' exists, the description narrows focus to package usage, providing decent differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description only explains what the tool does but does not provide guidance on when to use it versus alternatives like analyze, generate, etc. It lacks any 'when not to use' or context for selection, leaving the agent without decision support.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_submoduleA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the tool generates build instructions including backup and verification to prevent recovery failure, but does not detail side effects, permissions, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement, usage context, parameter list, and return description. It is appropriately sized, though some redundancy exists.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (4 parameters, no output schema, no annotations), the description covers purpose, usage context, and return format adequately. However, it omits error conditions, prerequisites, and differentiation from sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, and the description compensates by briefly explaining each parameter with examples. However, the explanations are not exhaustive and lack constraints or validation details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool generates a submodule stub for indirectly excluding C extension packages that cannot be directly stubbed. It specifies the use case and resource, but does not explicitly differentiate from the sibling 'generate' tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description instructs to use this tool based on 'submodule_stub_hints' from the 'analyze' tool, providing clear context. However, it lacks explicit exclusion criteria or alternatives for when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generateA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses key behaviors: generates based on symbol references, writes no files, returns a dictionary. With no annotations, description carries full burden and does so adequately, though could mention error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Moderate length with a logical structure: purpose, process, return format, parameters. No extraneous text, but could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, process, parameters, and return values. Without output schema, it lists return keys (files, referenced_symbols, stub_total_bytes), providing sufficient context for a generate tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage; the description provides clear explanations for all three parameters in an 'Args' section, fully compensating for missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it generates minimal stub code for a stubbable package, specifying verb and resource. However, does not explicitly differentiate from sibling tool 'generate_submodule', leaving some ambiguity about scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like 'analyze' or 'generate_submodule'. The description is purely functional, lacking context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• graphA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that it returns nodes classified by module type and includes statistics, which is helpful. However, it does not mention whether the tool is read-only, any side effects, authentication needs, or performance characteristics. With no annotations provided, the description carries the full burden but falls short of full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with an overview, then Args and Returns sections. It is relatively concise, though the opening line could be more direct. Overall, it is well-organized and not verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with three parameters and no output schema, the description provides a reasonable overview of what the tool does and returns (nodes, edges, statistics). It could be more detailed about the exact fields in the output, but it is sufficient for an agent to understand the tool's basic functionality.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only titles and types with no descriptions (0% coverage). The description compensates fully by explaining each parameter: entry_point is the project entry point file path, python_path is for site-packages (auto-detected if empty), and max_depth is maximum depth with a default of 5. This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool constructs and visualizes an import graph from an entry point, returning dependency relationships as nodes and edges. It distinguishes itself from sibling tools (analyze, check, generate, generate_submodule) by focusing on dependency graph construction and visualization.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus its siblings (analyze, check, generate, generate_submodule). It lacks explicit usage context, making it harder for an agent to decide which tool to invoke.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyzeA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully handles behavioral disclosure: it explains the analysis process and classification logic. However, it omits details like error handling, performance, or whether it modifies files, which would be needed for a perfect score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear introduction, bullet points for classifications, and an Args/Returns section. It is appropriately sized, though a slightly more concise phrasing could improve it.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a tool with no output schema: it details the return value (classified package list) and the three possible classifications, providing sufficient context for agent invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides clear semantics for all three parameters in the 'Args' section, explaining their purpose and defaults (e.g., max_depth default 10). This fully compensates for the 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool analyzes import graphs from an entry point and detects stubbable packages, with specific classification outcomes. This distinct purpose differentiates it from siblings like 'generate' or 'graph'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by explaining classifications and recommending exclusions for nofollow packages, but does not explicitly state when to use this tool vs. alternatives or provide context for when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.