Server Quality Checklist

Tool Scores

• fast_checkB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'Fast check' (implying speed) and 'Raises: RuntimeError: If check fails' (indicating error handling), but lacks details on performance characteristics (e.g., rate limits), authentication needs, or what constitutes a 'fail.' This leaves significant gaps for a tool with mutation-like implications (checking might involve external calls).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose. The 'Args:' and 'Returns:' sections add structure without redundancy. However, the 'Raises:' section, while useful, could be integrated more seamlessly, and the overall text is slightly verbose for such a simple tool, preventing a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no output schema, no annotations), the description is minimally complete. It covers the purpose, parameter, return type, and error handling, but lacks details on the 'Fast check result JSON' structure or behavioral traits like idempotency or side effects. This makes it adequate but with clear gaps for informed usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context beyond the input schema, which has 0% coverage. It specifies that 'url' is a 'URL to be verified,' clarifying the parameter's purpose and format (a string representing a URL). Since there's only one parameter, this adequately compensates for the schema's lack of descriptions, though it doesn't detail validation rules (e.g., URL format requirements).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Fast check if a URL has already been scanned.' It specifies the verb ('check') and resource ('URL'), and distinguishes it from siblings like 'new_scan' (which would scan) and 'get_scan' (which might retrieve detailed results). However, it doesn't explicitly differentiate from 'search' (which might also check for scanned URLs), keeping it from a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: use this for a quick verification of prior scanning, as opposed to 'new_scan' for initiating a scan. However, it lacks explicit guidance on when to use this versus 'get_scan' or 'search' (e.g., for speed vs. detail), and no exclusions or prerequisites are mentioned, making it only adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_scanB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds some context: it specifies the source ('urlDNA'), indicates potential failure modes ('Raises: RuntimeError: If fetch scan fails'), and notes the return format ('Truncated scan result JSON'). However, it lacks details on authentication needs, rate limits, or what 'truncated' entails, which are gaps for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the core purpose stated first, followed by structured sections for Args, Returns, and Raises. Each sentence earns its place by providing essential information, though the 'Raises' section could be slightly more detailed without sacrificing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (1 parameter, no output schema, no annotations), the description is somewhat complete but has gaps. It covers the basic purpose, parameter semantics, and error handling, but lacks usage guidelines, full behavioral context (e.g., auth), and details on the 'truncated' return format. This is adequate but not fully comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning beyond the input schema, which has 0% description coverage. It explains that 'scan_id' is 'The unique identifier of the scan', clarifying its purpose and format (a string ID). This compensates well for the schema's lack of descriptions, though it doesn't cover all potential nuances like ID format examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get scan results from urlDNA using the scan ID.' It specifies the verb ('Get'), resource ('scan results'), and source ('urlDNA'), which is clear and specific. However, it doesn't explicitly differentiate from sibling tools like 'fast_check' or 'search', which might also retrieve scan-related data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'fast_check' or 'search'. It mentions using a 'scan ID' but doesn't clarify prerequisites (e.g., that a scan must already exist from 'new_scan') or exclusions. This leaves the agent without context for tool selection among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• new_scanB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds some context: it's a submission tool that waits for results and can raise RuntimeError on failure. However, it lacks details on permissions, rate limits, timeouts, or what 'truncated' means in the return value, leaving gaps for a mutation-like operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the core purpose stated first. The Args, Returns, and Raises sections are structured but slightly verbose for a single parameter; every sentence adds value, though it could be more streamlined.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (submission with waiting, potential errors) and no annotations or output schema, the description is minimally adequate. It covers the basic operation and error handling but lacks details on output format beyond 'truncated scan result JSON,' which is vague, and doesn't address sibling tool relationships.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful semantics beyond the input schema, which has 0% coverage. It explains that the 'url' parameter is 'URL to submit for scanning,' clarifying its purpose. Since there's only one parameter, the baseline is high, and this extra detail compensates well for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Submit a URL to urlDNA and wait for the scan result.' It specifies the action (submit and wait) and resource (URL to urlDNA). However, it doesn't explicitly differentiate from sibling tools like 'fast_check' or 'get_scan', which likely have related scanning functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'fast_check' or 'get_scan'. It mentions waiting for results, which implies this might be a synchronous or blocking operation, but doesn't clarify use cases, prerequisites, or exclusions compared to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the tool returns a 'List of scan' and raises a 'RuntimeError' on failure, which adds some context. However, it lacks details on permissions, rate limits, pagination, or what happens in edge cases (e.g., no results). For a search tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose. Each section (searchable fields, operators, examples, args, returns, raises) adds value without redundancy. However, the 'Raises' section could be integrated more smoothly, and the structure is slightly fragmented with bullet points and separate headings.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a search tool with 1 parameter, no annotations, and no output schema, the description is moderately complete. It covers the query syntax and examples well but lacks details on output format (beyond 'List of scan'), error handling specifics, or integration with sibling tools. For a tool with no structured output, more guidance on return values would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds substantial meaning beyond the input schema, which has 0% coverage and only lists 'query' as a string. It explains that the query uses 'urlDNA CQL syntax,' lists searchable fields (e.g., domain, ip), supported operators (e.g., =, LIKE), and provides examples. This compensates well for the low schema coverage, though it could clarify the exact syntax format or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Search scans using urlDNA custom search syntax,' which is a specific verb (search) and resource (scans). It distinguishes itself from siblings like 'fast_check,' 'get_scan,' and 'new_scan' by focusing on query-based searching rather than quick checks, retrieval, or creation. However, it could be more precise by explicitly mentioning it searches through existing scans in a database or system.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by providing examples and searchable fields, suggesting it's for querying scans based on specific criteria. However, it lacks explicit guidance on when to use this tool versus alternatives like 'fast_check' (likely for quick checks) or 'get_scan' (likely for retrieving a single scan by ID). No exclusions or prerequisites are mentioned, leaving the agent to infer context from the examples.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.