Server Quality Checklist

Tool Scores

• shodan_searchC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions what information is returned (services, vulnerabilities, geographic distribution, country-based statistics) but doesn't address rate limits, authentication requirements, pagination behavior, or whether this is a read-only operation. The description provides some output context but misses key operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with three sentences that each add value. It's front-loaded with the core purpose, followed by return details, and ending with capabilities. No wasted words, though it could be slightly more structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 2 parameters and no output schema, the description provides adequate context about what the tool does and what information it returns. However, it lacks details about authentication, rate limits, and differentiation from sibling tools, which would be valuable given the server has multiple related lookup tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description doesn't add any meaningful parameter semantics beyond what's in the schema - it doesn't explain query syntax, advanced filter formats, or provide examples of valid queries.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches Shodan's database for internet-connected devices and returns detailed information about them. It specifies the resource (Shodan's database) and verb (search), but doesn't explicitly differentiate from sibling tools like 'ip_lookup' or 'dns_lookup' which might have overlapping functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions 'advanced search filters' but provides no guidance on when to use this tool versus alternatives like 'ip_lookup' or 'cve_lookup'. There's no mention of prerequisites, limitations, or specific scenarios where this tool is preferred over siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dns_lookupA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions batch resolution and the return format (IP addresses mapped to hostnames), but doesn't cover important aspects like rate limits, authentication requirements, error handling, or whether this is a read-only operation. The description provides basic behavior but lacks critical operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences that are front-loaded with core functionality, followed by batch capability, and ending with return format. Every sentence adds value with zero wasted words. The structure efficiently communicates the tool's purpose and capabilities.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no annotations and no output schema, the description provides adequate basic information about what the tool does and what it returns. However, it lacks important contextual details about authentication, rate limits, error cases, and specific usage scenarios that would help an agent use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single 'hostnames' parameter. The description adds marginal value by mentioning 'batch resolution of multiple hostnames' which reinforces the array nature of the parameter, but doesn't provide additional syntax, format, or constraint details beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Resolve domain names to IP addresses'), identifies the resource ('using Shodan's DNS service'), and distinguishes from siblings by focusing on forward DNS resolution (unlike reverse_dns_lookup or IP-based tools like ip_lookup). It provides a complete picture of what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for DNS resolution tasks and mentions batch capability, but doesn't explicitly state when to use this tool versus alternatives like reverse_dns_lookup or ip_lookup. No guidance on prerequisites, limitations, or specific scenarios where this tool is preferred over others is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cves_by_productA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: filtering capabilities (KEV, EPSS, date ranges), pagination support, and return content ('detailed vulnerability information including severity scores and impact assessments'). However, it omits details like rate limits, authentication needs, error handling, or whether it's read-only/destructive, leaving gaps for a mutation-sensitive agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with three sentences: purpose, features, and return value. It is front-loaded with the core purpose, and each sentence adds useful information without redundancy. Minor improvement could be made by integrating features more seamlessly, but it's efficient overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 9 parameters, no annotations, and no output schema, the description provides a good overview of functionality and return content. However, it lacks details on behavioral aspects like rate limits or error cases, and does not fully compensate for the absence of output schema, leaving the agent uncertain about the exact response structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 9 parameters. The description adds minimal value beyond the schema by mentioning filtering and sorting options (e.g., 'KEV status', 'EPSS score') without providing additional syntax or format details. This meets the baseline of 3 when schema coverage is high.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search for vulnerabilities affecting specific products or CPEs') and resource ('CVEs'), distinguishing it from siblings like cve_lookup (likely general CVE lookup) and cpe_lookup (CPE-specific). The verb 'search' with the qualifier 'affecting specific products or CPEs' provides precise scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through 'Supports filtering by KEV status, sorting by EPSS score, date ranges, and pagination,' suggesting when to use it for filtered searches. However, it lacks explicit guidance on when to choose this tool over alternatives like cve_lookup or cpe_lookup, and does not mention exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cpe_lookupA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions pagination support and the ability to return full details or just counts, which adds useful context beyond basic functionality. However, it does not cover aspects like rate limits, authentication needs, error handling, or response format details, leaving gaps in behavioral understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in three sentences: the first states the core functionality, the second adds key features (pagination and output options), and the third provides usage context. Each sentence earns its place without redundancy, making it front-loaded and appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, no output schema, no annotations), the description covers core functionality and some behavioral aspects but lacks details on response format, error cases, or integration with sibling tools. It is adequate as a minimum viable description but has clear gaps in providing a complete operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all parameters. The description adds value by explaining the purpose ('Search for CPE entries by product name') and the utility ('Useful for identifying specific versions and configurations'), which provides semantic context beyond the schema's technical details. Since no parameters are left undocumented, this exceeds the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search for Common Platform Enumeration entries by product name'), identifies the resource ('in Shodan's CVEDB'), and distinguishes it from siblings by focusing on CPE lookup rather than CVE lookup or other Shodan functions. It explicitly mentions what it returns ('full CPE details or just the total count'), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context ('Useful for identifying specific versions and configurations of software and hardware') but does not explicitly state when to use this tool versus alternatives like 'cve_lookup' or 'cves_by_product'. It mentions pagination and count options, which provide some guidance on functionality, but lacks direct comparisons or exclusions for sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reverse_dns_lookupA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the batch capability, return format (all known hostnames), and handling of missing results. However, it doesn't mention rate limits, authentication needs, or error conditions that would be important for a network tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with three sentences that each add value: states the core purpose, describes batch capability, and explains return behavior. No wasted words and front-loaded with the main functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a network lookup tool with no annotations and no output schema, the description provides adequate basic information but lacks details about response format structure, error handling, and operational constraints that would be important for complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'ips' parameter. The description adds context about batch lookups and what the parameter represents, but doesn't provide additional syntax or format details beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('perform reverse DNS lookups', 'find hostnames') and resources ('IP addresses'). It distinguishes from sibling tools like 'dns_lookup' and 'ip_lookup' by specifying the reverse direction and batch capability.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (to find hostnames from IP addresses with batch support). It doesn't explicitly mention when not to use it or name alternatives, but the context is sufficient for basic differentiation from siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cve_lookupA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the comprehensive data returned (CVSS scores, EPSS, KEV status, etc.), which adds useful context about output behavior, but does not mention rate limits, authentication needs, or error handling, leaving gaps for a tool with external API dependencies.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first states the purpose and source, and the second enumerates the returned details. Every sentence adds essential information with no wasted words, making it highly concise and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (querying external vulnerability data) and lack of annotations and output schema, the description does well by detailing the comprehensive return data. However, it could improve by mentioning response format or error scenarios, slightly limiting completeness for an API-dependent tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the single parameter 'cve' well-documented in the schema. The description does not add any parameter-specific details beyond what the schema provides, such as examples or edge cases, so it meets the baseline for high schema coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Query detailed vulnerability information') and resource ('from Shodan's CVEDB'), distinguishing it from sibling tools like cpe_lookup or cves_by_product by focusing on individual CVE details rather than product-based queries or broader searches.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying it queries 'detailed vulnerability information' for a CVE identifier, but does not explicitly state when to use this tool versus alternatives like cves_by_product or shodan_search. It provides clear intent but lacks explicit comparison or exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ip_lookupA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by disclosing behavioral traits: it specifies what information is returned (geolocation, open ports, services, SSL certificates, etc.), mentions conditional returns ('when present'), and indicates cloud provider detection. However, it doesn't cover rate limits, authentication needs, or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and efficiently lists return details in a single, well-structured sentence. Every phrase adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description provides good context about return types and conditions. It could be more complete by mentioning response format or error handling, but it adequately covers the tool's scope for a lookup operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single 'ip' parameter. The description doesn't add parameter-specific details beyond implying the IP address is used for comprehensive lookup, maintaining the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Retrieve comprehensive information') and resource ('about an IP address'), distinguishing it from sibling tools like dns_lookup or reverse_dns_lookup by focusing on IP-specific intelligence rather than DNS resolution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by listing the types of information returned (geolocation, services, etc.), but doesn't explicitly state when to use this tool versus alternatives like shodan_search or when not to use it (e.g., for domain-based queries).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.