Server Quality Checklist

Tool Scores

• search_conceptsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses the matching algorithm (BM25 + embeddings fused by reciprocal rank), that results are tagged as 'generated' provenance, and that empty results include an 'unmappable' explanation. This is solid behavioral disclosure, though it omits any explicit read-only or safety statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, then adds relevant behavioral details, usage guidance, and structured parameter/return docs. Every sentence contributes useful information without repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the minimal schema, no annotations, and output-schema presence, the description covers the essential invocation and interpretation details: search semantics, caveats about generated provenance, result behavior on no match, and parameter meanings. The sibling list further helps an agent distinguish this search tool from exact-lookup and hierarchy tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema is minimal, only marking query as required with no property definitions. The description fully compensates by explaining query as a free-text search string with an example and k as the maximum number of results to return, making both parameters unambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: searching ICD-10-CM concept names with a free-text clinical term or phrase. It doesn't explicitly contrast with siblings like lookup_concept, but the emphasis on free-text and paraphrase matching makes the tool's role clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use the tool: when the user has loosely-worded clinical descriptions and when exact terminology is unknown, saying 'always prefer this over guessing at exact terminology yourself.' It doesn't name alternatives or exclusion cases, but the usage context is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• expand_hierarchyA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries full behavioral disclosure. It warns that returned concepts are tagged 'generated' provenance, are structural rather than clinically validated, and should be treated as ungrounded until human confirmation. It also states the empty-set behavior for nonexistent codes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well structured and front-loaded: purpose, concrete example, critical provenance caveat, then parameter/return documentation. Every sentence adds value and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no annotations, the description covers the purpose, the seed_code semantics, the return type and behavior, and an important caveat about generated concepts. This is complete enough for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines seed_code as a string with 0% coverage, so the description must compensate. It does so by specifying 'An exact ICD-10-CM code' and providing the E11 example, communicating the precision and format expectations. It could add period-format guidance, but it is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a precise action and resource: 'Expand an ICD-10-CM code to every code beneath it in the billing hierarchy.' The E11 to E11.21 example makes the operation concrete and distinguishes it from sibling tools like lookup_concept or search_concepts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The use case is clearly implied: call this when you need all descendants of an exact ICD-10-CM code. The description does not explicitly name alternatives or exclusion conditions, so it falls short of a perfect 5, but the framing leaves little ambiguity about when it applies.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_conceptA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states exact-match behavior, the return type (Concept or None), and the absence of a match returning None. While it doesn't discuss edge cases like malformed codes, the core behavior is transparent for a lookup tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: purpose, usage guidance, parameter definition, and return behavior are each given in clear, non-redundant sentences. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter, the sibling context, and the presence of an output schema, the description is complete. It tells the agent when to use it, which sibling to use instead, what input to provide, and what to expect in return. Nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates by defining concept_code as 'an exact ICD-10-CM code' with a concrete example ('E11.21'). This adds meaningful semantics beyond the bare string type in the schema, though it doesn't specify formatting or normalization rules.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states a specific verb and resource: 'Look up a single ICD-10-CM concept by its exact billing code.' It also explicitly differentiates from search tools, making it clear this is an exact-code lookup rather than a free-text query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this when the caller already has a specific code' and contrasts with 'search_concepts' for free-text clinical terms. This provides clear when-to-use and when-not-to-use guidance, directly routing the agent to the correct sibling tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_curated_definitionA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it delivers: it discloses provenance semantics (curated vs generated), safety of using matches as-is, the limited bundled dataset, and the exact empty-set behavior with an 'unmappable' entry when no match exists or the library has not been fetched.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence earns its place: purpose, usage priority, fallback behavior, dataset limitation, and a clean param/returns/rtype breakdown. The most important routing guidance is front-loaded in the first two paragraphs.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter search tool with an output schema, the description covers all essential context: what it searches, when to use it, what it returns, how it signals failure, and a critical caveat about the bundled library. No important gap remains for an agent to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It defines the single parameter as 'Free-text population description' and supplies concrete examples like 'type 2 diabetes' and 'diabetic ketoacidosis,' making the expected input clear. It stops short of additional constraints or formatting details, but for one simple parameter this is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Search the OHDSI Phenotype Library for a validated cohort definition.' It further differentiates itself from siblings by positioning this tool as the curated-definition search and naming search_concepts/expand_hierarchy as the fallback alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage guidance is explicit and actionable: 'Use this FIRST for any population description that plausibly matches a peer-reviewed phenotype' and 'Only fall back to search_concepts/expand_hierarchy if nothing matches.' It also warns that a miss does not mean no curated definition exists, preventing a common misinference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.