Server Quality Checklist

Tool Scores

• verifyA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses a critical behavioral trait: the tool only reports provably absent references, so a clean result is meaningful and a block is never a guess. It also states the output (blocked references and real API), providing transparency beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded: the first sentence captures the main purpose, and the second paragraph adds brief usage guidance. Every sentence earns its place with no repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema or annotations, the description covers the essential context: what it does, what it returns, when to run it, and its certainty guarantee. It does not discuss error handling or prerequisite conditions, but for the tool's complexity, this is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds minimal new parameter-level meaning: 'Python source' maps to code, and 'this project' loosely ties to path, but this does not go beyond what the schema already provides for each parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Check Python source for references that provably do not exist') and specifies the exact scope: invented packages, invented methods, and missing helpers. This clearly distinguishes it from sibling tools (surface, context) by focusing on validation/verification rather than exploration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: 'Run this on code you are about to write.' It also explains when results are meaningful ('a clean result is meaningful'). However, it does not explicitly contrast with sibling tools or when not to use it, so it misses the 'when-not' aspect.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• surfaceA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the burden. It discloses how data is obtained ('Reads the installed package or parses the project file, so the answer is always current') and cost characteristics ('Cheap enough to re-ask'). It does not mention potential import side effects or failure modes, but provides meaningful behavioral context for a read-only introspection tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded. The first sentence defines the tool; the second sentence gives usage guidance and cost rationale. Every clause adds value, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema is provided, but the description states the essential return contents ('every public name with its real signature and a one-line summary') and the source ('installed package or project file'). It omits precise return format and error behavior, but this is adequate for a 4-parameter introspection tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions cover 100% of parameters, so the baseline is 3. The description does not add parameter-specific detail beyond what the schema already documents for target, limit, private, and summaries.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a concrete verb and resource: 'Get the exact API of a Python module, class, or project file' and specifies the output as 'every public name with its real signature and a one-line summary.' This clearly defines what the tool does and distinguishes it from generic file-reading or verification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Use this INSTEAD OF reading a file when you need to know how to call something.' It also gives a cost-benefit rationale ('5-10x fewer tokens', 'Cheap enough to re-ask') that helps the agent decide when to invoke it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• contextA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It discloses key behaviors: the closure is computed from code, follows project-local imports transitively, is budgeted, and is cheaper than reading files. It does not mention return format or failure modes, but the core behavioral traits are well conveyed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two tight paragraphs, front-loaded with the core purpose and then usage guidance. Every sentence contributes meaning—purpose, when to use, what it computes, cost efficiency, and refresh advice. No filler or repetition of schema fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity and no output schema, the description adequately explains what the result contains (file API + imported module surfaces). It lacks explicit detail about return formatting or edge cases (e.g., missing imports), but for an MCP tool definition it provides enough for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and each parameter has a description, providing a baseline of 3. The description adds value by explaining the 'budgeted block' concept and how depth ('followed transitively') and paths ('files you are about to edit') fit into the closure computation, going beyond the raw schema text.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool computes a dependency closure: 'the file's own API plus the exact surface of every module it imports, followed transitively through project code.' This is a specific verb+resource+scope and distinguishes it from generic file-reading tools. It also contrasts itself with 'similarity search,' further clarifying its unique purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'Call this INSTEAD OF reading a file and its dependencies, before you start editing.' It also advises refreshing the result rather than holding it in context. This clearly tells the agent when to use the tool and what it replaces, though sibling tools (surface, verify) are not directly named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.