Server Quality Checklist

Tool Scores

• analyze_ssis_packageA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes the output content but lacks behavioral details such as whether it's a read-only analysis, if it modifies the package, performance characteristics, error handling, or authentication needs. This is inadequate for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that efficiently lists all key output components without waste. It is front-loaded with the core action and resource, making it easy to scan and understand.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description partially compensates by detailing the report contents. However, it lacks information on behavioral traits, error cases, or output structure, leaving gaps for a tool with two parameters and no structured output documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters fully. The description does not add any meaning beyond what the schema provides, such as explaining parameter interactions or usage examples. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('analyze') and resource ('a single SSIS package (.dtsx file)'), and distinguishes it from siblings by specifying it analyzes a single package rather than scanning multiple packages (scan_ssis_packages) or converting/deploying (convert_ssis_package, deploy_to_adf).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by listing what the tool returns, but does not explicitly state when to use it versus alternatives like scan_ssis_packages (for multiple packages) or validate_adf_artifacts (for ADF validation). No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• convert_ssis_packageA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it generates multiple JSON artifacts and stubs, returns a summary and warnings, and supports specific features like Self-Hosted IR and cross-package dedup. However, it lacks details on error handling, performance, or side effects like file system changes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by a concise list of generated artifacts and supported features. Every sentence adds value without redundancy, making it efficient and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (13 parameters, no annotations, no output schema), the description is reasonably complete. It covers the conversion process, outputs, and key features, but lacks details on return values (beyond 'summary' and 'warnings') and error conditions, which could be important for a tool with many parameters and no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 13 parameters thoroughly. The description adds no additional parameter semantics beyond what the schema provides, such as explaining interactions between parameters or usage examples. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Convert a single SSIS package') and resource ('.dtsx file') to output ('Azure Data Factory JSON artifacts'), listing the exact types generated. It distinguishes from sibling tools like analyze_ssis_package or deploy_to_adf by focusing on conversion rather than analysis or deployment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives like analyze_ssis_package or validate_adf_artifacts. It mentions the tool's capabilities but does not specify prerequisites, ideal scenarios, or when other tools might be more appropriate, leaving usage context implied at best.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_adf_artifactsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes what the tool does (validates for structural correctness) and what it returns (list of issues or success message), but doesn't mention error handling, performance characteristics, permission requirements, or whether it modifies files. It provides basic behavioral context but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with zero waste: first states purpose, second specifies validation criteria, third describes return behavior. Every sentence earns its place by adding distinct information. The description is appropriately sized and front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter validation tool with no annotations and no output schema, the description provides adequate context about what it validates and what it returns. It could be more complete by specifying validation error formats or success message structure, but covers the essential functionality given the tool's relative simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single parameter. The description adds minimal value beyond what the schema provides, only reinforcing that it's for 'generated ADF JSON artifacts' without adding format details or constraints. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('validate'), target resource ('ADF JSON artifacts in a directory'), and scope ('structural correctness'). It distinguishes from siblings by focusing on validation rather than analysis, conversion, deployment, or scanning operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (when you need to check ADF JSON artifacts for structural issues) but doesn't explicitly state when to use this tool versus alternatives. No guidance is provided about when NOT to use it or what specific scenarios warrant validation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_ssis_packagesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the discovery action and return format, but doesn't mention important behavioral aspects like whether this is a read-only operation, potential performance impacts for large directories, authentication requirements for git/SQL sources, or error handling. The description adds basic context but lacks comprehensive behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that each earn their place: the first states the core purpose and output, the second provides critical constraint information. It's front-loaded with the main functionality and wastes no words on redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, multiple source types) and no annotations or output schema, the description is adequate but incomplete. It covers the basic purpose and constraints but lacks information about authentication needs, error conditions, performance characteristics, and what 'basic metadata' includes. For a discovery tool with multiple source types, more behavioral context would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema already fully documents all 4 parameters. The description adds minimal value beyond the schema by mentioning the source_type enum values and that it discovers '.dtsx files', but doesn't provide additional semantic context about parameter interactions or usage patterns beyond what's in the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('discover all SSIS packages'), target resource ('.dtsx files'), and output format ('JSON list of found packages with name, path, and basic metadata'). It distinguishes itself from siblings like analyze_ssis_package (which analyzes rather than discovers) and convert_ssis_package (which converts rather than discovers).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool by specifying the source_type options and what it returns. However, it doesn't explicitly state when NOT to use it or mention alternatives among the sibling tools (e.g., use analyze_ssis_package for detailed analysis instead of just discovery).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deploy_to_adfA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and excels by disclosing critical behavioral traits: the deployment order (linked services → datasets → data flows → pipelines → triggers), that triggers are deployed in Stopped state requiring manual activation, and the authentication method (DefaultAzureCredential with specific options). This goes well beyond basic functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly front-loaded with the core purpose in the first sentence, followed by essential behavioral details. Every sentence adds value: deployment order, trigger state, and authentication method. There's zero wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex deployment tool with no annotations and no output schema, the description is highly complete—covering purpose, behavior, and authentication. It lacks only minor details like error handling or response format, which would be needed for a perfect score given the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents all 5 parameters. The description doesn't add any parameter-specific meaning beyond what's in the schema (e.g., it doesn't clarify artifact formats or directory structure). This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Deploy ADF JSON artifacts') and resource ('to an Azure Data Factory instance'), with precise scope ('from a local directory'). It distinguishes from sibling tools like 'validate_adf_artifacts' by focusing on deployment rather than validation or SSIS-related tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (deploying artifacts to ADF) and implies an alternative through the 'dry_run' parameter for validation without deployment. However, it doesn't explicitly state when NOT to use it or compare it to other deployment methods, keeping it from a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.