Server Quality Checklist

Tool Scores

• generate_reportA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the tool runs audit_contract (implying a computational step) and returns both Markdown and structured findings. However, it does not discuss potential side effects, rate limits, or performance implications of running the audit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences that efficiently convey the action, input, and output. Every word is useful; no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with three parameters and no output schema, the description adequately explains the main purpose and return values. It mentions both rendered Markdown and structured findings, but omits details about the structure of those findings. Given the simplicity of the tool, this is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with all three parameters already described in the schema. The description adds no additional meaning beyond what the schema provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a full audit_contract and renders the result as a clean Markdown report. It specifies both the action ('run and render') and the resource ('audit_contract result'), and distinguishes from siblings like audit_contract (which likely only runs the audit without rendering).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for obtaining a formatted report, but does not explicitly state when to use this tool versus alternatives like audit_contract or check_vulnerability. No when-not or alternative guidance is provided, though the purpose is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_vulnerabilityA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Lists return fields (vulnerability status, severity, SWC id, fix recommendation) but does not disclose if it reads state, requires authentication, or handles errors. With no annotations, the description carries the burden but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences: first defines purpose and output, second gives usage recommendation. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequate for a simple check tool with 2 parameters, 100% schema coverage, and no output schema. Lacks error handling info but is otherwise complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions cover 100% of parameters. Description adds value by giving concrete examples of vulnClass values and labeling source as Solidity code, beyond schema metadata.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'targeted check for a single vulnerability class' and provides specific examples (reentrancy, integer-overflow, etc.), distinguishing it from sibling tools like audit_contract.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly recommends use for 'quick spot-checks during development', implying it is not for comprehensive audits. Context with sibling tools reinforces differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gas_optimizationA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description fully bears the burden. It details the output (ordered list with estimated savings) and lists optimization types (storage packing, loop optimizations, etc.). This goes beyond basic purpose, though it doesn't mention side effects or limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no fluff. It front-loads the action and includes relevant details about the output. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description covers the main functionality and output characteristics. It could mention output format or version constraints, but it is still fairly complete and distinct from sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter 'source', so baseline is 3. The description adds context by indicating the source is reviewed for gas savings and that the output is a list of suggestions, enhancing understanding of the parameter's role.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a gas optimization review of a Solidity source file and returns ordered suggestions. It is specific about the resource (Solidity source) and distinguishes from sibling tools like audit_contract (broader audit) and check_vulnerability (security).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for gas optimization but does not explicitly state when to use this tool over siblings or when not to use it. An agent can infer context from sibling names, but no direct guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_contractA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It transparently describes the multi-pass process and specifies return values (severity, SWC id, etc., risk score 0-100, summary). No mention of side effects or destructive behavior, but the tool is read-only by nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: process, outputs, usage. Front-loaded with key information, no redundant words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and moderate complexity, it covers the audit process and outputs well. Minor gaps: no mention of runtime, limits, or error handling, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage on both parameters, and description adds practical context: source accepts pastes, file dumps, multi-contract files; contractName inferred when omitted. This goes beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool performs a 3-pass audit of a Solidity source file (recon, deep scan, severity scoring) and returns structured findings plus risk score. This distinguishes it from siblings like check_vulnerability (single check), gas_optimization, and generate_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this as the default entry point when reviewing a contract,' providing clear guidance on when to use this tool. Lacks explicit when-not or alternatives, but the strong recommendation suffices.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.