Server Quality Checklist

Tool Scores

• get_owasp_top10C
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves information (implying a read-only operation) but doesn't address critical aspects like whether it requires authentication, rate limits, error handling, or the format/scope of returned data. For a tool with no annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without unnecessary words. It is front-loaded with the core purpose and appropriately sized for a simple tool with one optional parameter, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete for effective use. It doesn't explain what information is returned (e.g., vulnerability details, descriptions, mitigations), how results are structured, or any behavioral constraints. For a tool with no structured data to supplement it, the description should provide more context to be fully actionable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal value beyond the input schema, which has 100% coverage. It mentions 'for a specific year or category,' hinting at the optional 'category' parameter's purpose, but doesn't provide additional context like valid year ranges or category examples beyond what's in the schema. With high schema coverage, the baseline is 3, and the description doesn't significantly enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get information about') and resource ('OWASP Top 10 vulnerabilities'), making it immediately understandable. It distinguishes the tool by specifying the domain (OWASP Top 10) but doesn't explicitly differentiate it from sibling tools like 'get_security_context' or 'search_security_docs', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings (get_security_context, list_security_sources, search_security_docs). It mentions the scope ('for a specific year or category'), but this is more about parameter usage than contextual alternatives. Without explicit when/when-not instructions or named alternatives, it falls short of providing meaningful usage guidelines.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_security_contextC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool 'Returns detailed information from multiple authoritative sources,' which gives some context about output richness, but doesn't cover important aspects like whether this is a read-only operation, potential rate limits, authentication requirements, or what format the information returns. For a tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences that directly address purpose and output characteristics. It's front-loaded with the main function. While efficient, it could potentially benefit from slightly more structure to separate purpose from behavioral context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no annotations, no output schema, and the description provides minimal behavioral context, the description is incomplete. It doesn't explain what 'comprehensive security context' entails, how the information is structured, or what users can expect from the 'multiple authoritative sources.' For a tool that presumably returns complex security information, this leaves significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the single parameter 'topic' well-documented in the schema. The description adds no additional parameter information beyond what's already in the schema. According to the scoring rules, when schema coverage is high (>80%), the baseline is 3 even with no param info in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('comprehensive security context'), and specifies the scope ('for a specific topic'). However, it doesn't explicitly differentiate from sibling tools like 'get_owasp_top10' or 'search_security_docs', which might also provide security information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings. It mentions 'multiple authoritative sources' but doesn't specify when this comprehensive approach is preferred over the more focused 'get_owasp_top10' or broader 'search_security_docs'. No exclusions or alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_security_sourcesA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool lists sources and categories, but does not describe return format (e.g., structure, pagination), performance characteristics, or any constraints (e.g., rate limits, authentication needs). This leaves significant gaps for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('List all available security documentation sources and their categories') with zero redundant information. Every word contributes directly to understanding the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate for basic understanding but incomplete for operational use. It lacks details on return values (since no output schema exists) and behavioral traits, which are critical for an agent to invoke it effectively in context with sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and schema description coverage is 100% (empty schema). The description appropriately does not discuss parameters, as none exist. It focuses on the tool's purpose, which is sufficient given the parameterless design.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List') and resource ('all available security documentation sources and their categories'), distinguishing it from sibling tools like get_owasp_top10 (specific standard), get_security_context (context retrieval), and search_security_docs (search functionality). It precisely defines the tool's scope without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for obtaining a comprehensive overview of security sources, but provides no explicit guidance on when to use this tool versus alternatives like search_security_docs for filtered results or get_security_context for contextual information. It lacks explicit when-not-to-use statements or prerequisite conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_security_docsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes the tool's function (searching security docs with natural language) and scope (specific sources and content types), but does not disclose behavioral traits such as rate limits, authentication requirements, response format, or error handling. The description is accurate but lacks operational details needed for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first defines the tool's purpose and scope, and the second provides usage instructions. Every sentence adds essential information without redundancy, making it front-loaded and appropriately sized for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (search across multiple sources), lack of annotations, and no output schema, the description is partially complete. It covers the purpose, scope, and basic usage but omits details on behavioral traits, response format, and error handling. This is adequate as a minimum viable description but has clear gaps for a search tool with no structured output information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters (query, limit, source) thoroughly. The description adds minimal value beyond the schema by implying the 'query' parameter accepts natural language and listing possible sources for filtering, but does not provide additional syntax, format details, or examples. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search across security documentation') and identifies the comprehensive scope of sources (OWASP, NIST, AWS, etc.) and content types (guidance, best practices, vulnerabilities, controls). It distinguishes itself from sibling tools like 'get_owasp_top10' (specific to OWASP) and 'list_security_sources' (listing rather than searching) by emphasizing broad, multi-source search capabilities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('Use natural language queries to find relevant security guidance...'), but it does not explicitly state when not to use it or name specific alternatives. It implies usage for broad searches across multiple sources, which differentiates it from more focused siblings, but lacks explicit exclusions or direct comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.