Server Quality Checklist

Tool Scores

• identify_business_rulesC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description only states a read-like action ('extract') without detailing side effects, authentication needs, error handling, or return behavior. The description does not compensate for missing annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no wasted words, but it is too minimal given the number of parameters and absence of schema descriptions. It could be expanded without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, the description does not explain return values. It omits parameter details and usage context, making it incomplete for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the description adds no information about the three parameters (source_code, max_rules, api_key). Their purpose, format, and behavior are entirely undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool extracts IF/EVALUATE business rules from COBOL PROCEDURE DIVISION. It uses a specific verb ('Extract') and resource, and distinguishes from sibling tools like parse_cobol_program and plan_migration_phases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. Does not mention prerequisites, limitations, or which scenarios are appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_test_harnessC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; the description lacks details on side effects, dependencies, or output characteristics, leaving the agent uninformed about the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, front-loaded with the core purpose, but too terse for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 undocumented parameters, no annotations, and the presence of an output schema not described, the description fails to adequately inform the agent about proper usage and expected results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not explain any of the three parameters (source_code, target_stack, api_key), providing no semantic value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'generate' and the resource 'skeleton test harness' with context 'parallel-run equivalence testing', distinguishing it from sibling tools focused on migration tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives, no prerequisites or conditions mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• estimate_migration_complexityC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fails to disclose behavioral traits such as authentication needs, side effects, or external dependencies. It only states the action without any behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise, but it sacrifices necessary detail for brevity. It could be more structured without becoming overly long.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema and three parameters, the description covers only a basic action. It lacks information about output, parameter usage, and expected input format, making it incomplete for reliable tool selection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning only for the target_stack parameter by listing examples, but ignores source_code and api_key entirely. Given 0% schema description coverage, this is insufficient compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'estimate' and the resource 'migration effort', and lists the supported target stacks, making it distinct from sibling tools like parse_cobol_program or plan_migration_phases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus alternatives, nor does it mention prerequisites or exclusions. The context is implied but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• parse_cobol_programB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states parsing and extraction but omits side effects, authentication requirements (api_key is unexplained), and whether the tool modifies anything or is read-only. The security and behavioral context is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the verb and resource. It is efficient with no wasted words, though it omits important details about parameters and usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (two simple parameters, output schema exists), the description is moderately complete. It covers the main purpose and outputs but misses critical context about the api_key parameter and usage constraints. The presence of an output schema reduces the need to describe return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description offers no explanation for either parameter. The required 'source_code' is implied but not explicitly linked; the optional 'api_key' is completely unexplained. The description adds no semantic value beyond the schema field names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: parsing a COBOL source file. It lists specific extraction targets (divisions, paragraphs, file IO, SQL/CICS, copybooks, complexity), making the purpose concrete. The sibling tools have distinct purposes (estimation, test generation, etc.), so differentiation is clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when parsing COBOL code, but does not explicitly state when to use this tool versus siblings (e.g., for analyzing complexity vs. using estimate_migration_complexity). No scenarios, prerequisites, or limitations are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• plan_migration_phasesB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description only states the basic action without behavioral context (e.g., whether it is read-only, requires authentication, has side effects). The api_key parameter suggests authentication, but not explained. Leaves agent uninformed about side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence with no redundancy. Efficiently conveys core purpose, though could further structure to highlight inputs and output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters and output schema, description covers the main function but lacks details on parameter usage for api_key, error conditions, and any constraints. Adequate for a simple tool but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%. Description adds meaning for loc and complexity by linking them to the plan, but does not explain their semantics (e.g., units, range). The api_key parameter is not mentioned, leaving its purpose unclear. Description partially compensates but incomplete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the verb 'Generate', the resource 'migration plan', and the inputs 'LOC + complexity'. It distinguishes itself from sibling tools like estimate_migration_complexity by focusing on plan generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. No mention of prerequisites or exclusions. Siblings like estimate_migration_complexity might be used for input estimation, but the description does not address this.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.