Server Quality Checklist

Tool Scores

• search_host_vulnsB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It mentions results are scoped to the configured team, but omits details like read-only nature, pagination behavior, or rate limits. Adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no redundancy. Could front-load the scope ('Results are scoped to the configured team') more prominently, but overall efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 11 parameters and no output schema, more context is needed—e.g., default sorting, error handling, or relationship to other data. Description covers only basic purpose and scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage, so baseline is 3. Description adds no extra meaning beyond the schema; it only restates purpose and return fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool searches for host vulnerabilities and lists the return fields (triage level, status, affected host/package, solution, advisory). The term 'host vulnerabilities' differentiates it from siblings like search_app_vulns or search_container_vulns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. Does not mention when-not to use or provide context for choosing it over other search tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_app_vulnsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so description carries full burden. It reveals return fields and team scoping but omits whether the tool is read-only, authentication needs, rate limits, or side effects. For a search tool, this is minimal transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences with no redundant fluff. Every sentence adds value: purpose+returns, scoping.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 10 optional parameters and no output schema, the description covers key returned fields and scoping. It lacks details on default pagination behavior (though schema mentions page/size) and the structure of results, but overall is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so each parameter has a description. The tool description adds no extra meaning beyond schema; it only provides high-level context. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for app library vulnerabilities from yamory and lists returned fields (triage, status, package, solution, CVE). It implicitly distinguishes from siblings like 'search_host_vulns' by specifying 'app library'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this vs sibling tools (e.g., search_vulns). The mention of team scoping implies a use-case, but no when-not-to or alternatives are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_asset_vulnsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explicitly notes that solution/fixedVersion are not available and directs to the detail URL for remediation, which is useful. It also mentions scoping to the team. However, it omits authentication needs, rate limits, or any destructive/read-only nature beyond what's implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences long, front-loaded with the core purpose, and every sentence adds necessary information: what it returns, a critical limitation, and scope. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 10 parameters with full schema descriptions and no output schema, the description covers the key behavioral aspects (return fields, missing fields, scoping). It omits details like pagination behavior or default ordering, but these are partially covered by schema. Overall, it is sufficiently complete for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 10 parameters have descriptions in the schema (100% coverage), so the baseline is 3. The description adds value by explaining the return fields and a specific limitation (missing solution/fixedVersion), but it does not enrich parameter-level semantics beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search IT asset vulnerabilities detected by yamory', specifying the resource (IT asset vulnerabilities), source (yamory), and scope (team). It also lists returned fields, distinguishing it from sibling tools like search_app_vulns or search_host_vulns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for IT asset vulnerabilities within a configured team, but it does not explicitly state when to use this tool over siblings or provide exclusion criteria. No alternatives or when-not-to-use guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_container_vulnsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. The description mentions scoping to team and return fields, but lacks details on behavioral traits like pagination, authentication, or rate limits. As a search tool, it's minimally adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that front-load the purpose and key return fields. No superfluous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 11 parameters, no output schema, and 100% schema coverage, the description adequately explains the tool's purpose and return content, though it omits explanation of pagination behavior beyond the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameters are well described in the schema. The description adds no additional meaning beyond what is already in the schema, earning a baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it searches container image vulnerabilities from yamory, listing specific return fields. This distinguishes it from sibling tools targeting app, asset, host vulns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for container image vulnerabilities via its name and description, but no explicit when-to-use or alternatives guidance beyond the sibling tool names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_vulnsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that results are grouped by source type and scoped to the configured team, which is useful context. However, it does not mention pagination behavior or what happens when filters yield no results.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with four sentences, each serving a clear purpose: explaining what it does, recommending its use, describing output grouping, and noting scoping. No redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the 10 parameters and no output schema, the description covers high-level behavior (grouping, scoping, recommended use) but lacks a quick note on combining filters or pagination details, which would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description does not add any parameter-specific meaning beyond what the schema already provides, which is adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches vulnerabilities across app libraries and container images, distinguishing it from sibling tools like search_app_vulns. It specifies the verb 'search' and resource 'vulnerabilities' with additional context on grouping and scoping.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'This is the recommended default tool for general vulnerability queries,' providing clear usage guidance. While it doesn't explicitly state when not to use it, the sibling tool names imply alternatives for specific scopes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.