Server Quality Checklist

Tool Scores

• get_fix_commandsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the critical non-execution behavior upfront ('WITHOUT executing them') and clarifies that the output is a list of commands, implying no side effects. It could add more context (e.g., whether network access or dependency installation is needed), but the key safety-relevant trait is well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The key differentiator ('WITHOUT executing them') is front-loaded, followed by the usage context and the return-summary grouping. Every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity (3 params, no output schema), the description adequately covers what the tool returns (grouped commands), when to use it (before apply_fixes), and the critical no-execution guarantee. It could mention edge cases like empty results or required project discovery details, but for a preview/list tool this is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage for all three parameters, so the description need not restate them. The schema already explains path, packages (optional filter), and severity_filter (default all). The description adds no parameter-level meaning beyond what the schema provides, which matches the baseline of 3 for complete schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') and a precise resource ('safe upgrade commands for vulnerable packages'), then immediately distinguishes the tool from its likely sibling apply_fixes by stating 'WITHOUT executing them'. It also names the grouped-by-ecosystem output, so an agent can clearly identify the tool's role in the workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to use this tool ('Use this to show the user what will be changed before calling apply_fixes'), naming the alternative and the sequencing. It does not address scan_project or open_dashboard, but the main decision point between preview and execution is clearly covered.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_projectA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states the tool scans the project, auto-detects multiple manifest formats, queries live OSV.dev data, and returns a structured vulnerability report with severity counts, risk score, and fix recommendations. It does not explicitly state whether the scan modifies files, which would be useful for a tool that may be followed by apply_fixes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three focused sentences. The first states the core action, the second lists capabilities and data source, and the third gives usage ordering. Every sentence earns its place with no repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description provides a solid high-level picture: input is a project directory, it detects manifests, queries OSV.dev, and returns a structured report with specific components. It lacks an explicit statement about side effects or network requirements, but the visible detail is sufficient for an agent to call it correctly as a first step.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage of the three parameters, so the baseline for value-add is 3. The description reinforces that path refers to a project directory and mentions OSV.dev, which relates to the offline parameter, but it adds no parameter-level detail beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description has a specific verb and resource: 'Scan a project directory for CVE vulnerabilities.' It also differentiates itself from siblings by stating this is the first step before open_dashboard or apply_fixes. The tool's role as a scanner that produces a report is unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to use this tool: 'Use this as the first step before open_dashboard or apply_fixes.' This names alternatives and establishes context. However, it does not mention when not to use it or describe any exclusions for the sibling get_fix_commands, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• open_dashboardA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and adds valuable behavioral context: it discloses that this launches a browser dashboard for human review, returns a URL, and handles existing instances by returning the existing URL if already running. However, it doesn't mention potential side effects like opening browser tabs or resource usage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by workflow guidance and behavioral details, all in three efficient sentences with zero wasted words, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description does well by explaining the tool's purpose, usage context, and return value (URL). However, it lacks details on error handling or what happens if the path is invalid, leaving minor gaps in completeness for a tool with two parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters (path and port). The description adds no additional parameter semantics beyond what's in the schema, such as format examples or constraints, meeting the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Launch the osv-ui visual dashboard in the browser') and resource ('for human review'), distinguishing it from siblings like apply_fixes, get_fix_commands, and scan_project by emphasizing it's the 'HUMAN-IN-THE-LOOP step' for review before applying fixes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit guidance on when to use this tool ('always offer this before applying fixes') and distinguishes it from alternatives by positioning it as a prerequisite step to apply_fixes, with clear context about its role in the workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• apply_fixesA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states this is a 'DESTRUCTIVE action that modifies package files,' which is crucial for a mutation tool. It also describes the return behavior ('Returns the command output for each fix applied') and includes a safety note about user confirmation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by critical warnings and usage guidelines. Every sentence adds value: the first states the action, the second highlights destructiveness and prerequisites, and the third explains the return. There is no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive tool with no annotations and no output schema, the description is largely complete: it covers purpose, behavioral risks, usage workflow, and return format. However, it could slightly enhance completeness by mentioning potential side effects (e.g., system downtime) or error handling, though the current level is adequate given the schema coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description does not add specific meaning beyond the schema (e.g., it doesn't elaborate on 'path' or 'packages' formats). However, it reinforces the importance of explicit packages ('Must be explicit — never fix all without user confirmation'), which aligns with but doesn't exceed schema guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Execute package upgrade commands') and resource ('to fix CVEs'), distinguishing it from siblings like 'get_fix_commands' (which only retrieves commands) and 'scan_project' (which only identifies issues). It goes beyond the tool name 'apply_fixes' by specifying the exact mechanism.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'ALWAYS call get_fix_commands first and confirm with the user before calling this.' This directly addresses when to use this tool versus the sibling 'get_fix_commands' and includes a prerequisite workflow. The description also implies when not to use it (e.g., without confirmation).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.