Server Quality Checklist

Tool Scores

• get-notificationsC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions the tool provides 'high level summary, only include the summary, not the raw data' which gives some behavioral context about output format. However, it doesn't disclose important behavioral traits like whether this is a read-only operation, potential rate limits, authentication requirements, or what happens with pagination (implied by 'limit' and 'page' parameters).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, reasonably concise sentence that front-loads the core purpose. However, it could be more structured by separating the 'what' from the 'how' - currently it mixes functional description ('Get all notifications data') with implementation guidance ('be sure to present the data in a way that is easy to understand').

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 6 parameters, no annotations, no output schema, and true nested objects in the schema, the description is insufficient. It doesn't explain the relationship between the filtering parameters and the 'high level summary' output, doesn't describe the return format, and provides no guidance on error conditions or limitations. The description focuses on presentation aspects while neglecting operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents all 6 parameters. The description adds no additional parameter semantics beyond what's in the schema descriptions. It doesn't explain how parameters interact with the 'high level summary' output or provide usage examples. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Get all notifications data from OnSecurity' which provides a clear verb+resource combination, but it doesn't distinguish this tool from its siblings (get-blocks, get-findings, etc.) beyond mentioning 'notifications'. The description adds that it provides 'high level summary' data, which helps differentiate its output format from raw data retrieval tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its sibling tools (get-blocks, get-findings, get-prerequisites, get-rounds). It mentions presenting data 'in a way that is easy to understand for the client' which implies a presentation-focused use case, but doesn't specify when this summary view is preferable to raw data access or when to choose notifications over other data types.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-roundsC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool gets data and should output a summary, but lacks critical details: it doesn't mention whether this is a read-only operation, potential rate limits, authentication requirements, or what happens with large datasets (e.g., pagination behavior). The instruction to 'present the data in a way that is easy to understand' is more about output formatting than tool behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise with three sentences, but the second sentence about 'When replying...' contains presentation instructions that don't belong in a tool description (should be in the agent's prompt, not the tool definition). This reduces efficiency. The structure is front-loaded with the core purpose but includes extraneous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (8 parameters, no annotations, no output schema), the description is incomplete. It doesn't address key behavioral aspects like safety (read vs. write), performance considerations, or what the output looks like. The presentation instructions don't compensate for missing tool behavior context. For a data retrieval tool with rich parameters but no structured safety hints, more guidance is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with all 8 parameters well-documented in the input schema. The description adds no parameter-specific information beyond what's already in the schema. According to the scoring rules, when schema coverage is high (>80%), the baseline is 3 even with no param info in the description, which applies here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves 'all rounds data from OnSecurity' and provides a 'high level summary', which is a specific verb+resource combination. It distinguishes the types of rounds (pentest, scan, radar) but doesn't explicitly differentiate from sibling tools like get-findings or get-blocks, keeping it at a 4 rather than 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like get-findings or get-blocks. It mentions that the output should be a summary for the client, but this is about presentation rather than tool selection criteria. No explicit when/when-not or alternative tools are referenced.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-prerequisitesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states this is a read operation ('Get'), but doesn't mention authentication needs, rate limits, pagination behavior (beyond what parameters imply), error conditions, or response format. For a tool with 6 parameters and no annotation coverage, this leaves significant gaps in understanding how the tool behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences. The first sentence states the core purpose, and the second provides helpful domain context about what prerequisites are. There's no wasted text, though it could be slightly more structured for readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, nested objects, no output schema, and no annotations), the description is insufficient. It doesn't address authentication, response format, error handling, or practical usage scenarios. The agent would need to infer much from the parameter schema alone, which is inadequate for a tool with this level of functionality.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 6 parameters. The description adds no parameter-specific information beyond implying round_id is needed for filtering. This meets the baseline of 3 when the schema does the heavy lifting, but doesn't provide additional semantic context like example values or usage patterns.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('prerequisites data from OnSecurity for a specific round'), and explains what prerequisites are ('requirements that need to be fulfilled before a security assessment can begin'). However, it doesn't explicitly differentiate this tool from its siblings (get-blocks, get-findings, etc.), which all appear to be data retrieval tools for different entities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools, nor does it specify prerequisites for use (e.g., authentication requirements, round existence). The only implied context is needing a round_id, but this is covered in the required parameter.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-blocksB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions that blocks are 'reusable security finding templates' and that usage counts reflect 'most common findings ACROSS ALL CLIENTS', which adds useful context about data scope and aggregation. However, it doesn't address critical behavioral aspects like whether this is a read-only operation, potential rate limits, authentication requirements, pagination behavior (implied by 'limit' and 'page' parameters but not explained), or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise with three sentences that each add value: defining blocks, explaining their purpose, and noting the usage count feature. It's front-loaded with the core purpose ('Get all blocks data'). While efficient, the third sentence could be slightly more streamlined as it contains some redundancy ('across different assessments' and 'across pentests and scans').

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (10 optional parameters, no output schema, no annotations), the description provides adequate but incomplete context. It explains what blocks are and hints at usage analytics, but doesn't address the tool's behavioral characteristics, return format, or error handling. For a data retrieval tool with many filtering options, more guidance on result structure and limitations would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, providing detailed documentation for all 10 parameters. The description adds minimal parameter semantics beyond the schema, only implicitly referencing 'used_count' through the mention of 'how often a block is used'. Since schema coverage is high, the baseline score of 3 is appropriate as the description doesn't significantly enhance parameter understanding beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get all blocks data from OnSecurity' with a specific verb ('Get') and resource ('blocks data'). It explains what blocks are ('reusable security finding templates'), which helps distinguish them from sibling tools like 'get-findings' or 'get-rounds'. However, it doesn't explicitly differentiate this tool from its siblings in terms of when to use each.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some implied usage guidance by noting that blocks are 'the basis of findings across pentests and scans' and mentioning that 'you can get how often a block is used' to identify common findings across clients. However, it lacks explicit instructions on when to use this tool versus alternatives like 'get-findings' or 'get-rounds', and doesn't specify prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-findingsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it mentions the tool returns 'a high level summary, only include the summary, not the raw data', it doesn't describe important behavioral aspects like authentication requirements, rate limits, pagination behavior (implied by limit/page parameters but not explained), or what happens when no findings exist. The description adds some context about presentation format but misses key operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise but has structural issues. The first sentence contains multiple clauses that could be separated. The usage warning is tacked on at the end rather than integrated. While not excessively verbose, the flow could be improved for better front-loading of key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 9 parameters, no annotations, and no output schema, the description is incomplete. It doesn't explain the relationship between parameters, how filtering works in practice, what the summary format looks like, or error conditions. The description focuses on usage constraints and basic purpose but leaves many operational questions unanswered given the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 9 parameters thoroughly. The description only mentions the 'round_id' parameter specifically ('You can optionally filter findings by round_id') and implies summary formatting. It adds minimal value beyond what the schema provides, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get all findings data from OnSecurity from client in a high level summary'. It specifies the verb ('Get'), resource ('findings data'), and scope ('high level summary'). However, it doesn't explicitly differentiate from sibling tools like 'get-blocks' beyond the usage warning at the end.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'ONLY USE THIS TOOL WHEN ASKED FOR FINDINGS RELATED TO A CLIENT OR MY FINDINGS, NOT THE BLOCKS TOOL.' This clearly states when to use this tool versus alternatives, specifically warning against using it for 'blocks' requests.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.