Server Quality Checklist

Tool Scores

• list_checks_by_account_typeA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries full burden. It only states the function and return format but discloses no behavioral traits such as ordering, pagination, error conditions, or permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Very concise (two sentences plus a Return line), front-loaded with purpose. Every sentence adds value; no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional parameter and a clear return type (dictionary), the description is nearly complete. It explains the filter and output format. Could mention that account_type defaults to 'all' (from schema), but that is minor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description adds allowed values ('application', 'audit', 'log-archive', 'management', 'all') that are not present in the input schema (no enum). This adds meaningful constraint beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Get all checks filtered by account type' which uses a specific verb and resource, and distinguishes from siblings like list_checks_by_service (filter by service) and describe_check (describe single check).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives like list_checks_by_service or describe_check. The description only lists allowed values for account_type but does not provide context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_servicesA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries full burden. It states it returns a dictionary but does not disclose that it is a read-only operation, any authentication requirements, or potential side effects. Minimal behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, no wasted words. Perfectly concise for a simple tool with no parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema details, the description is adequate but not thorough. While the tool is simple, missing information like whether the list is static or if any permissions are needed lowers completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist. The description confirms return type ('Dictionary'). Schema coverage is 100%, and the description adds value by explaining the output, justifying a baseline 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Get all services supported by SRA Verify,' which is a specific verb+resource combination. It clearly distinguishes from sibling tools that deal with checks, not services.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. However, since sibling tools are about checks, the usage context is implied. A score of 3 reflects adequate but missing explicit recommendations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_checkA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description should disclose behavioral traits. It minimally mentions that the tool returns a dictionary with details, but does not address authentication, rate limits, side effects, or whether it is safe/read-only. This is insufficient for a read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise: one line summary, followed by Args and Returns sections. Every sentence adds value, and the structure is clear and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists, the description adequately summarizes the return fields (name, service, etc.) without needing to restate the schema. For a simple tool with one parameter, it covers the essential information, though it could reference the output schema for exact structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only specifies 'check_id' as a string with a title. The description adds significant value by providing an example format ('SRA-GUARDDUTY-01'), which helps agents understand the expected format. Schema description coverage is 0%, so the example compensates well.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'Get detailed information' and the resource 'specific check'. It distinguishes itself from sibling tools like 'list_checks_by_account_type' and 'run_check' by focusing on a single check's details.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a check_id and need details, but it does not explicitly state when to use this tool versus alternatives or mention any exclusions. The example format provides some guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_checks_by_serviceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It describes the return format as a dictionary mapping check IDs to check information, but it does not mention potential side effects, authentication needs, or rate limits. The tool is a simple list operation, so the description is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with a single-sentence purpose followed by short args and returns sections. It is front-loaded and free of unnecessary words, earning its place with every sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (single parameter, output schema provided), the description adequately covers purpose, parameter semantics, and return structure. However, it lacks guidance on error conditions or usage context, which would make it fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only the parameter name and type with no description. The description adds meaning by stating 'Filter by service name (e.g., 'GuardDuty', 'CloudTrail', 'IAM')', giving examples that help the agent understand valid values. This compensates for the zero schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get all checks filtered by service', which is a specific verb and resource. It further specifies the filtering criterion, distinguishing it from sibling tools like 'list_checks_by_account_type' and 'describe_check'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when needing checks for a specific service but does not explicitly state when to use this tool versus alternatives like 'list_checks_by_account_type'. There is no guidance on when not to use it or mention of prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_checkA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description discloses return format (Dict with findings/summary or error), parameter behaviors (e.g., region defaults to all), and conditional requirements for audit/log accounts. It does not explicitly state read-only or destructive nature, but the context of a security check implies mutation is unlikely.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with clear Args and Returns sections. Every sentence adds value without redundancy, and the structure facilitates quick scanning for key details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters, conditional requirements, and an output schema, the description covers the tool's behavior and return values well. Minor gap: it could mention prerequisites like permissions or that some checks may require specific account types, but sibling tools handle listing details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description fully explains each parameter: check_id with example, audit_accounts/log_archive_accounts with context, role_arn as optional, region with default behavior. This adds significant meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs 'Run a security check and return results,' which is a specific verb and resource. It distinguishes from siblings that describe, list, or manage checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus siblings like describe_check or list_checks. The usage is implied by the tool's action, but no when-not-to-use or alternative suggestions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.