Server Quality Checklist

Tool Scores

• adv_mark_false_positiveC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action ('Mark') but doesn't explain what this entails—whether it's a write operation, requires permissions, affects the finding's status permanently, or has side effects. This is a significant gap for a mutation tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste—it directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a mutation tool (marking findings) with no annotations and no output schema, the description is incomplete. It doesn't cover behavioral aspects like permissions, side effects, or what happens after marking, nor does it explain the return value. This leaves gaps that could hinder correct tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters thoroughly. The description adds no additional meaning beyond what the schema provides, such as explaining parameter interactions or usage examples. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Mark') and resource ('a finding') with the specific action ('as a false positive'), making the purpose unambiguous. However, it doesn't explicitly differentiate from its sibling 'adv_unmark_false_positive' beyond the opposite action, missing a direct comparison that would elevate it to a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'adv_unmark_false_positive' or other scanning tools. It lacks context about prerequisites (e.g., after a scan) or exclusions, leaving the agent to infer usage based on the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• adv_scan_fileB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'session-aware analysis' and LLM configuration, but doesn't cover critical aspects like whether this is a read-only or destructive operation, authentication requirements, rate limits, error handling, or what the scan results look like. For a security scanning tool with 8 parameters, this leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences that efficiently convey the core functionality and a key behavioral aspect. It's front-loaded with the main purpose. However, the second sentence about 'session-aware analysis' could be more clearly integrated or expanded for better flow.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex security scanning tool with 8 parameters, no annotations, and no output schema, the description is incomplete. It doesn't explain what the tool returns, how results are structured, error conditions, or important behavioral constraints. The mention of 'Clean Architecture' and 'session-aware analysis' adds some context but doesn't compensate for the missing information needed for effective tool use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 8 parameters thoroughly. The description doesn't add any parameter-specific information beyond what's in the schema. It mentions 'Clean Architecture' and 'session-aware analysis' which provide some context but don't directly explain parameter usage. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Scan a file for security vulnerabilities using Clean Architecture.' It specifies the action (scan), target (file), and domain (security vulnerabilities). However, it doesn't explicitly differentiate from sibling tools like 'adv_scan_code' or 'adv_scan_folder' beyond mentioning 'file' versus 'code' or 'folder' in their names.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some implied usage context: 'Automatically uses session-aware analysis when LLM is configured.' This suggests when LLM features might be relevant. However, it lacks explicit guidance on when to use this tool versus alternatives like 'adv_scan_code' or 'adv_scan_folder', and doesn't mention prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• adv_scan_folderB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'session-aware project analysis' and implies automation with LLM configuration, but fails to describe critical behaviors such as whether the scan is destructive, what permissions are required, how results are returned (e.g., format, persistence), or any rate limits. This leaves significant gaps for a security scanning tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences that efficiently convey the core functionality and a key behavioral aspect. It's front-loaded with the main purpose, though it could be slightly more structured by explicitly mentioning the tool's scope relative to siblings.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex security scanning tool with 8 parameters, no annotations, and no output schema, the description is incomplete. It lacks details on behavioral traits (e.g., safety, permissions), output handling, and differentiation from sibling tools, making it inadequate for full contextual understanding by an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, providing clear documentation for all 8 parameters. The description adds no additional parameter semantics beyond what's in the schema, such as explaining interactions between parameters (e.g., how 'use_llm' and 'use_validation' relate). Given the high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Scan a directory for security vulnerabilities using Clean Architecture.' It specifies the verb ('scan'), resource ('directory'), and methodology ('Clean Architecture'), but doesn't explicitly differentiate it from sibling tools like 'adv_scan_code' or 'adv_scan_file' which likely perform similar scanning on different targets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage context with 'Automatically uses session-aware project analysis when LLM is configured,' which implies when LLM analysis is enabled. However, it doesn't offer explicit guidance on when to choose this tool over alternatives like 'adv_scan_code' or 'adv_scan_file,' nor does it specify prerequisites or exclusions for use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• adv_get_statusB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states what information is retrieved but doesn't describe how the data is formatted, whether it's real-time or cached, potential rate limits, authentication requirements, or error conditions. For a status tool with zero annotation coverage, this leaves significant gaps in understanding its operational behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Get comprehensive server status') and lists key components without redundancy. Every word earns its place, making it easy for an agent to parse quickly. There's no wasted verbiage or structural issues.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (low, with no parameters) and lack of annotations/output schema, the description is minimally adequate. It covers what the tool does but lacks behavioral details (e.g., response format, performance characteristics). For a status tool, more context on data freshness or structure would be helpful, but it meets the basic threshold for a simple read operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (empty schema). The description doesn't need to explain parameters, so it appropriately focuses on the tool's purpose. A baseline of 4 is justified since no parameter documentation is required, and the description doesn't introduce unnecessary parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('comprehensive server status'), specifying three components: session management capabilities, active sessions, and cache statistics. It distinguishes from siblings like 'adv_get_version' by focusing on operational status rather than version information. However, it doesn't explicitly differentiate from other status-related tools (none exist in the sibling list), so it's not a perfect 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, timing considerations, or comparisons to sibling tools like 'adv_scan_code' or 'adv_scan_file'. The agent must infer usage from the purpose alone, which is insufficient for optimal tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• adv_get_versionB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. While 'Get' implies a read operation, the description doesn't disclose any behavioral traits - no information about authentication requirements, rate limits, error conditions, or what format the version information returns. For a tool with zero annotation coverage, this is insufficient disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise at just three words. It's front-loaded with the essential information - what the tool does - with zero wasted words. Every word earns its place in communicating the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 0-parameter read operation with no output schema, the description is minimally complete. It tells what the tool does but lacks important context about return format, authentication, or differentiation from similar tools. Without annotations or output schema, the agent has incomplete information about what to expect from this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the baseline is 4. The description appropriately doesn't discuss parameters since none exist, and the schema already documents this completely. No additional parameter information is needed or provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Get server version information' clearly states the verb ('Get') and resource ('server version information'), making the purpose immediately understandable. However, it doesn't distinguish this from sibling tools like 'adv_get_status' - both appear to retrieve server information but for different aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With siblings like 'adv_get_status' that also retrieve server information, the agent has no indication whether this should be used for version checking, system monitoring, or other contexts. No prerequisites or exclusions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• adv_scan_codeB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'session-aware analysis' and 'project context,' which adds some behavioral context, but it doesn't describe critical aspects like authentication requirements, rate limits, error handling, or what the scan results look like. For a security scanning tool with no annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with zero waste. It's front-loaded with the core purpose and efficiently adds context about session-aware analysis. Every sentence earns its place by providing essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a security scanning tool with 7 parameters, no annotations, and no output schema, the description is incomplete. It lacks details on behavioral traits (e.g., what the tool returns, error cases), doesn't differentiate from siblings, and provides minimal usage guidance. The description should do more to compensate for the missing structured data.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description doesn't add any parameter-specific information beyond what's in the schema, such as explaining interactions between parameters (e.g., how use_semgrep and use_llm work together). Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Scan code content for security vulnerabilities using Clean Architecture.' It specifies the action (scan), target (code content), and objective (security vulnerabilities). However, it doesn't explicitly differentiate from sibling tools like adv_scan_file or adv_scan_folder, which appear to be related scanning tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage context: 'Automatically uses session-aware analysis with project context when available.' This implies the tool leverages existing context, but it doesn't explicitly state when to use this tool versus alternatives like adv_scan_file or adv_scan_folder, nor does it provide exclusions or prerequisites for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• adv_unmark_false_positiveB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action ('Remove') but doesn't explain what happens after unmarking (e.g., does the finding reappear in reports, is it reversible, are permissions required?). This leaves critical behavioral traits like mutation effects and security implications unclear, making it inadequate for a tool that modifies data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with zero wasted words, front-loading the core action. It efficiently communicates the essential purpose without redundancy or unnecessary elaboration, making it highly concise and well-structured for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity as a mutation operation with no annotations and no output schema, the description is insufficient. It lacks details on behavioral outcomes, error handling, or return values, leaving gaps in understanding how the tool functions in practice. For a tool that alters data, more context is needed to ensure safe and correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the input schema fully documents both parameters ('finding_uuid' and 'adversary_file_path'). The description adds no additional parameter semantics beyond what's in the schema, such as format details or examples. This meets the baseline for high schema coverage but doesn't enhance understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Remove') and target ('false positive marking from a finding'), making the purpose immediately understandable. It distinguishes itself from siblings like 'adv_mark_false_positive' by specifying the opposite operation. However, it doesn't explicitly mention the resource type (e.g., security finding) or context, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a finding was previously marked as a false positive and needs to be reverted, but doesn't explicitly state when to use it versus alternatives. No guidance is provided on prerequisites, side effects, or when not to use it, leaving usage context partially inferred rather than clearly defined.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.