Server Quality Checklist

Tool Scores

• assess_publication_biasB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the methods used but doesn't disclose side effects, dependencies, expected return values, or any caveats about the input data. The absence of such information leaves the agent uncertain about the tool's behavior beyond its basic function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is concise and front-loaded with the core purpose. It avoids unnecessary wording and is easy to scan. However, it is perhaps too terse to provide meaningful behavioral or usage context, so it loses a point for being underspecified rather than merely concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a nested object parameter (analysis_results) and an optional methods array, but the description does not explain how to provide these, what a valid analysis_results object looks like, or what the tool returns. With no output schema, the description is insufficient to guide an agent through a complete workflow, especially given the complexity of the input.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds no additional meaning beyond the schema's parameter descriptions, but since the schema already explains each parameter adequately, it doesn't need to compensate. The description doesn't clarify the relationship between analysis_results and the methods, but this is a minor gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Perform publication bias assessment using funnel plots, Egger's test, and trim-and-fill method.' This specifies the verb (perform), the resource (publication bias assessment), and the methods involved, effectively distinguishing it from sibling tools like perform_meta_analysis or generate_forest_plot.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor any exclusions or prerequisites. It doesn't mention that publication bias assessment typically follows a meta-analysis, nor does it reference sibling tools or indicate the required input format for analysis_results.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_cochrane_reportB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosing behavior, but it only says 'generate'. It does not mention that a file is written to output_path, whether existing files are overwritten, or any side effects. The scope of 'comprehensive' remains undefined.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence front-loaded with the verb and object. Every word adds value, with no filler or repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has four parameters including a nested object, but the description gives no context on workflow positioning, expected report structure, or output behavior. The absence of an output schema makes the description the only narrative source, and it is too sparse to fully orient an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% coverage with descriptions for all four parameters, so baseline is 3. The description adds slight value by noting the report is in Cochrane format and mentions HTML/PDF, which aligns with the format enum, but it does not enrich the meaning of analysis_results or output_path beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (generate), the object (comprehensive meta-analysis report), and the format (Cochrane format, HTML/PDF). It is distinct from sibling tools like perform_meta_analysis or generate_forest_plot, which cover other stages of the workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus siblings. It does not mention prerequisites (e.g., meta-analysis results must be ready) or exclude use for tasks like plotting or validation. The intended context is implied only by the name and sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_forest_plotB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only states the purpose and that output follows Cochrane standards, but does not mention overwriting behavior, required input structure, or any side effects. This is insufficient for a tool without annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that communicates the core purpose effectively. It is front-loaded and free of fluff, though it could include a brief note on expected input from perform_meta_analysis to improve usefulness without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool generates a plot file, but the description omits what the return value is (or if it just saves to output_path), what data structure analysis_results must have, and how this fits with sibling tools. Given no annotations and no output schema, the description should provide more context to be complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds no additional meaning to the parameters (e.g., what analysis_results should look like, how plot_style differs). Baseline of 3 is appropriate since the schema handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates a publication-ready forest plot following Cochrane standards. It uses a specific verb ('create'), a specific resource ('forest plot'), and adds a quality criterion ('publication-ready', 'Cochrane standards'), which distinguishes it from siblings like generate_cochrane_report or perform_meta_analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage in a Cochrane-style meta-analysis context, but does not explicitly state when to use this tool versus alternatives like generate_cochrane_report. No when-not-to-use guidance or alternative tools are mentioned, so it relies on the user's understanding of the workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• perform_meta_analysisB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It only mentions the R packages and Cochrane Handbook, which provide methodological context but not behavioral traits such as whether the tool writes outputs, requires pre-validated data, or has side effects. The return value is also undisclosed, especially important given no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action, and contains no filler. The second sentence adds useful context (Cochrane Handbook) without being verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a complex tool with four parameters, no output schema, and no annotations. The description does not explain what the tool returns, how it connects to sibling tools (e.g., after validate_cochrane_data, before generate_forest_plot), or any limitations. The 'Validated study data' in the schema hints at prerequisites but the description does not elaborate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with each parameter (data, model, effect_measure, heterogeneity_test) having a clear description. The tool description adds no additional parameter semantics beyond the schema, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Execute meta-analysis using R metafor/meta packages.' This is a specific verb and resource, and it distinguishes this tool from its siblings (import_revman_data, validate_cochrane_data, generate_forest_plot, assess_publication_bias, generate_cochrane_report), which handle supporting tasks rather than the core analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through 'Follows Cochrane Handbook Chapter 10 guidelines,' suggesting this is a standard meta-analysis tool. However, it does not explicitly state when to use this tool versus alternatives like validate_cochrane_data or generate_forest_plot, nor does it provide exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• import_revman_dataA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral disclosure burden. It describes the parsing/extraction behavior and mentions adherence to 'Cochrane standards,' which adds context. However, it does not disclose side effects (e.g., whether data is persisted), permission requirements, or error handling behavior, leaving some ambiguity for an import tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences, front-loaded with the main action, and every word provides useful information. There is no redundancy or filler, making it highly concise while covering the essentials.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, no output schema) and lack of annotations, the description is moderately complete. It explains what data is parsed and extracted, but it does not clarify whether the tool returns a dataset, stores data, or requires any external context. For a tool likely used as the first step in a workflow, this leaves some ambiguity about its output or side effects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%: both 'file_path' and 'format' have clear descriptions in the schema. The tool description adds context about supported data structures ('structured Cochrane datasets and flat record arrays') but does not add meaning to individual parameters beyond the schema, so the baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Import and parse') and resource ('meta-analysis data'), and explicitly lists supported formats. It is well differentiated from sibling tools (validate, analyze, plot, bias, report), which focus on later workflow stages.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied — it is the data ingestion step before validation and analysis — but the description does not explicitly state when to use it over alternatives or when not to use it. The mention of 'structured Cochrane datasets and flat record arrays' provides some applicability guidance, but no direct exclusions or alternative references.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_cochrane_dataA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the checks performed, which is useful, but does not mention any side effects, output format, or whether data is modified. For a validation tool, read-only behavior is implied but not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action, and no wasted words. It is highly concise while conveying the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description should indicate what the tool returns (e.g., a validation report, pass/fail). It also doesn't explain differences between 'basic' and 'comprehensive' validation. However, it does mention the key checks, providing some completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description only mentions general validation areas, not the parameters themselves. It doesn't clarify what 'validation_level' does beyond the schema's brief 'Level of validation checks'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Validate') and resource ('study data'), and lists concrete checks (PICO criteria, sample sizes, effect sizes, data quality), clearly distinguishing it from sibling tools like import_revman_data or generate_forest_plot.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a validation workflow but does not explicitly state when to use this tool vs alternatives, nor provide prerequisites or exclusions. It lacks clear when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.