Server Quality Checklist

Tool Scores

• cost_effectiveness_modelA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but lacks critical behavioral details. It mentions analysis types but doesn't disclose whether this is a read-only or write operation, what permissions are needed, whether it's computationally intensive, or what happens to the output (e.g., where results are stored). The mention of 'project' parameter saving to a directory hints at persistence but isn't fully explained.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first states the core purpose and scope, the second enumerates specific analysis components. Every phrase adds value without redundancy, making it front-loaded and easy to parse despite the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with 13 parameters, nested objects, no output schema, and no annotations, the description provides good high-level context but leaves gaps. It explains the analytical approach but doesn't cover behavioral aspects like computational requirements, error handling, or output details. The absence of annotations increases the burden on the description, which it partially meets but not fully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant value beyond the 62% schema coverage by explaining the analytical framework (cost-utility analysis with specific components like ICER, QALY, PSA) and referencing guidelines. While it doesn't detail individual parameters, it provides crucial context about what the tool fundamentally does that the schema alone doesn't convey, though some parameter relationships remain implicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Build a cost-utility analysis') and the comprehensive scope of what it produces (ICER, QALY, PSA, sensitivity analysis, CEAC). It distinguishes itself from siblings by focusing on economic modeling rather than knowledge management or dossier preparation, with explicit mention of following ISPOR and NICE guidelines.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through its reference to specific guidelines (ISPOR, NICE) and analysis types, suggesting it's for health economic evaluations. However, it doesn't explicitly state when to use this tool versus alternatives like 'hta_dossier_prep' or provide clear exclusions or prerequisites for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• literature_searchA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool returns structured results with audit trails suitable for HTA submissions, which adds useful context about output quality. However, it lacks details on rate limits, authentication needs for enterprise sources, or potential costs/limitations of the search operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is overly verbose with a long list of sources in the first sentence, making it difficult to parse quickly. While informative, it could be more front-loaded and structured for clarity, with some details better placed in the schema or annotations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of the tool (6 parameters, no output schema, no annotations), the description is mostly complete by explaining the broad purpose, sources, and output suitability. However, it could better address behavioral aspects like authentication or limitations to fully compensate for the lack of annotations and output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema by implying the query parameter should target drug/indication evidence, but does not provide additional syntax or format details. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('search') and the comprehensive scope of resources (PubMed, ClinicalTrials.gov, bioRxiv/medRxiv, etc.), distinguishing it from sibling tools like cost_effectiveness_model or hta_dossier_prep. It explicitly mentions the purpose is to find evidence for drugs or indications with HTA suitability.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (searching evidence for drugs/indications with HTA submissions), but does not explicitly state when not to use it or name specific alternatives among siblings. It implies usage for evidence gathering versus other tools focused on modeling or dossier preparation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• knowledge_readA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the read-only nature implicitly through the verb 'Read' and specifies access limitations ('Only raw/ and wiki/ subtrees accessible'), but doesn't mention authentication requirements, rate limits, error conditions, or what happens with invalid paths. It adds some behavioral context but leaves important operational details unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences with zero waste. The first states the core purpose, the second clarifies path semantics, and the third specifies access limitations. Every sentence earns its place by adding distinct, necessary information. The description is appropriately sized and front-loaded with the main functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read operation with 2 parameters, 100% schema coverage, and no output schema, the description provides adequate but not complete context. It covers the what, where, and access limitations, but lacks information about return format, error handling, authentication needs, or performance characteristics. Given the simplicity of the tool and good schema coverage, this is minimally viable but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters fully. The description adds minimal value beyond the schema by clarifying that path is 'relative to project root' and providing example formats, but doesn't explain parameter interactions or constraints beyond what's in the schema descriptions. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Read a file') and resource ('from a project's raw/ or wiki/ tree'), with explicit scope limitations ('Only raw/ and wiki/ subtrees accessible'). It distinguishes from sibling tools like knowledge_write (write vs read) and knowledge_search (search vs direct read).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('Read a file from a project's raw/ or wiki/ tree') and path requirements ('Path is relative to project root'), but doesn't explicitly state when NOT to use it or name specific alternatives like knowledge_search for broader searching. The sibling tool list shows knowledge_search exists, but no direct comparison is made.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• knowledge_searchA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool returns file paths with line numbers and snippets, which adds useful behavioral context. However, it lacks details on permissions, rate limits, or error handling, leaving gaps for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by usage guidance, in two efficient sentences with zero wasted words, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description provides basic purpose and usage but lacks details on return format, error cases, or operational constraints. For a search tool with 5 parameters, this is adequate but has clear gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all parameters. The description adds no additional parameter semantics beyond what's in the schema, such as explaining search logic or result formatting, resulting in the baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('search a project's knowledge base') and resources ('raw/ and wiki/'), and distinguishes it from siblings like knowledge_read, knowledge_write, and literature_search by specifying it searches previously-retrieved content without external API calls.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool ('to find previously-retrieved literature, model runs, and compiled wiki content without re-querying external APIs'), which implicitly differentiates it from literature_search. However, it does not explicitly state when not to use it or name alternatives, keeping it from a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• knowledge_writeA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that the tool writes files and supports Obsidian-style wikilinks, which is useful behavioral context. However, it doesn't mention important behavioral aspects like whether this overwrites existing files, what permissions are required, error conditions, or response format. For a write operation with zero annotation coverage, this leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in three sentences: first states the core action and constraints, second provides usage context, third adds feature support. Every sentence adds value with zero wasted words, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a write tool with 3 parameters, 100% schema coverage, but no annotations and no output schema, the description provides adequate purpose and usage context. However, it lacks important behavioral details about mutation effects (overwrite vs create), error handling, and response format that would be needed for complete understanding. The absence of output schema means the description should ideally address what the tool returns.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters. The description adds some value by clarifying the path format ('MUST start with wiki/ and end with .md') and content capabilities ('Can include YAML frontmatter and [[wikilinks]]'), but doesn't provide significant additional semantic context beyond what the schema descriptions offer.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Write a file'), target resource ('to the project's wiki/ tree'), and distinguishes it from sibling tools like knowledge_read and knowledge_search by focusing on writing rather than reading/searching. It provides concrete purpose beyond just the tool name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('to compile/organize evidence from raw/ files into a structured knowledge base') and mentions path format requirements. However, it doesn't explicitly state when NOT to use it or name specific alternatives among siblings like knowledge_read for reading existing files.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• hta_dossier_prepA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the tool's core function (structuring evidence into submission formats) and output (draft sections with gap analysis), but doesn't mention important behavioral aspects like whether this is a read-only or write operation, what permissions might be needed, how long processing takes, or error handling. The description adds value but leaves significant behavioral questions unanswered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that each earn their place. The first sentence establishes the core function and scope, while the second specifies input requirements and integration points. There's zero wasted language, and the most important information (what the tool does) comes first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with 9 parameters, no annotations, and no output schema, the description is adequate but has clear gaps. It explains the tool's purpose and input relationships well, but doesn't describe the output format in detail (beyond 'draft sections with gap analysis'), doesn't mention error conditions or limitations, and leaves behavioral aspects unspecified. Given the complexity, more completeness would be expected.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 67% schema description coverage, the description compensates well by providing crucial context about parameter relationships and tool integration. The statement 'accepts output from literature_search and cost_effectiveness_model' clarifies the semantics of evidence_summary and model_results parameters, which is valuable information not captured in the schema descriptions alone. However, it doesn't explain all parameters beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('structure evidence', 'produces draft sections with gap analysis') and resources ('HTA body-specific submission format'). It explicitly distinguishes from sibling tools by mentioning it 'accepts output from literature_search and cost_effectiveness_model', showing it operates downstream of those tools rather than duplicating their functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool by specifying it processes outputs from two specific sibling tools (literature_search and cost_effectiveness_model). However, it doesn't explicitly state when NOT to use it or mention alternatives for similar formatting tasks, which prevents a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• project_createA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively reveals key traits: the idempotent nature ('Idempotent — returns existing project if already created'), the prerequisite requirement for other tools, and the initialization of specific resources. However, it doesn't mention potential side effects like file system changes or error conditions, leaving some behavioral aspects uncovered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by critical behavioral notes. Every sentence earns its place by providing essential information without redundancy. The structure is logical and efficiently conveys key points in minimal text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (initialization with idempotency) and the absence of both annotations and an output schema, the description does a good job covering purpose, usage, and key behavior. However, it lacks details on what the tool returns (since no output schema exists) and doesn't fully address all potential side effects or error cases, leaving some gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description doesn't add any meaningful semantic context beyond what's in the schema—it doesn't explain relationships between parameters or provide usage examples. This meets the baseline expectation when schema coverage is complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Initialize a new HEOR project workspace'), the resources involved ('directory skeleton and project.yaml metadata'), and distinguishes this tool from siblings by explaining its prerequisite role for using the 'project' parameter in other tools. It goes beyond a simple tautology by detailing what initialization entails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool ('Required before using the `project` parameter in other tools') and provides a clear alternative scenario ('returns existing project if already created'). This gives the agent precise guidance on timing and fallback behavior without misleading information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.