Server Quality Checklist

Tool Scores

• veracode_applicationsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'list and search' but doesn't describe key behaviors such as pagination handling (implied by page/size parameters), rate limits, authentication requirements, or the format of returned data. This leaves significant gaps for a tool with 5 parameters and no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core functionality ('List and search Veracode application profiles') with zero wasted words. It's appropriately sized for the tool's scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, no output schema, no annotations), the description is incomplete. It lacks details on return values, error handling, pagination behavior, and how it differs from sibling tools. With no annotations to fill gaps, this minimal description is insufficient for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, providing clear details for all 5 parameters (e.g., name filtering with partial match, policyCompliance and scanType enums, pagination defaults). The description adds no additional parameter semantics beyond what's in the schema, so it meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'List and search Veracode application profiles,' which includes both listing and searching functions. It specifies the resource (application profiles) and platform (Veracode), but doesn't differentiate from sibling tools like 'veracode_findings' or 'veracode_policy_compliance' that might also involve application data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'veracode_findings' for detailed findings or 'veracode_policy_compliance' for compliance checks, nor does it specify prerequisites or contexts for application profile listing versus searching.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• veracode_findingsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It implies a read operation ('Get') but doesn't specify if it's paginated (though parameters suggest it), rate-limited, requires authentication, or what the output format is. This is a significant gap for a tool with 9 parameters and no output schema, making it inadequate for safe and effective use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's front-loaded and wastes no space, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (9 parameters, no annotations, no output schema), the description is incomplete. It doesn't address behavioral aspects like pagination, error handling, or output structure, which are critical for a data retrieval tool. The high parameter count and lack of structured support make this description insufficient for reliable agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 9 parameters. The description adds no additional meaning beyond the schema, such as explaining interactions between parameters (e.g., 'severity' vs. 'severityGte') or usage notes. This meets the baseline for high schema coverage but doesn't enhance understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('vulnerability findings/flaws for a Veracode application'), making the purpose specific and understandable. However, it doesn't differentiate from sibling tools like 'veracode_policy_compliance' or 'veracode_scan_status' which might also relate to findings or application data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'veracode_applications' or 'veracode_policy_compliance'. It lacks context about prerequisites, such as needing an application GUID, or exclusions, like not using it for compliance checks. This leaves the agent with minimal usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• veracode_policy_complianceC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states what the tool does ('Check policy compliance status') but fails to describe key behaviors like whether this is a read-only operation, if it requires authentication, potential rate limits, or what the output format looks like. For a tool with no annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any wasted words. It is appropriately sized and front-loaded, making it easy for an agent to parse quickly and understand the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of checking policy compliance with 4 parameters and no output schema, the description is incomplete. It doesn't explain what the tool returns (e.g., a list of applications with compliance details), how pagination works with the 'page' and 'size' parameters, or behavioral aspects like error handling. This leaves gaps for an agent to correctly invoke and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so the schema fully documents all four parameters. The description adds no additional meaning beyond what's in the schema, such as explaining relationships between parameters or usage nuances. With high schema coverage, the baseline score of 3 is appropriate as the description doesn't compensate but also doesn't detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Check') and resource ('policy compliance status for Veracode applications'), making it immediately understandable. However, it doesn't explicitly distinguish this tool from its sibling tools like 'veracode_applications' or 'veracode_findings', which might also relate to compliance information, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, such as the sibling tools listed. It lacks context about prerequisites, typical use cases, or exclusions, leaving the agent to infer usage based on the tool name and parameters alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• veracode_scan_statusC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states it 'Get[s] scan status and summary report', implying a read-only operation, but doesn't clarify aspects like authentication requirements, rate limits, error handling, or what the 'summary report' entails. This leaves significant gaps for a tool that likely interacts with external APIs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is appropriately sized and front-loaded, though it could be slightly more structured by including key details like authentication or output format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of interacting with a Veracode API (implied by the tool name), no annotations, and no output schema, the description is incomplete. It lacks critical context such as authentication needs, rate limits, error scenarios, and details on the 'summary report' output. This makes it inadequate for an agent to use the tool effectively without additional assumptions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the single parameter 'applicationGuid' clearly documented in the schema. The description adds no additional meaning beyond what the schema provides, such as explaining the format of the GUID or where to find it. Given the high schema coverage, a baseline score of 3 is appropriate as the description doesn't compensate but doesn't need to.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'scan status and summary report for a Veracode application', making the purpose specific and understandable. However, it doesn't explicitly differentiate from sibling tools like 'veracode_applications' or 'veracode_findings', which might also retrieve application-related information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'veracode_applications' or 'veracode_findings'. It lacks context about prerequisites, such as needing an existing application, or exclusions, leaving the agent to infer usage based on the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.