Server Quality Checklist

Tool Scores

• gene_phenotype_detailB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, and idempotentHint, setting a baseline for safety. The description adds context beyond annotations: it explains why this is a tool (SQL double-counting) and describes the verdict types. This is helpful but not exhaustive—there is no mention of rate limits, authentication needs, or what happens if inputs are invalid. The description adds some behavioral insight but falls short of full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is verbose and includes extraneous technical reasoning about SQL double-counting and the tool's origin. While informative, the first three sentences could be condensed. The key actionable information (handling lists, verdict meanings) appears only in the last two sentences. The structure lacks front-loading of essential details, making it less efficient for an AI agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists (context confirms), the description does not need to detail return values. It explains the tool's purpose, input batch behavior, and verdict semantics, which suffices for most use cases. However, it omits specifics about supported biobanks, ancestry strata, or example phenotype IDs. The description is adequate for an expert but could be more complete for general AI consumption.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers all 5 parameters with descriptions (100% coverage), so the baseline is 3. The description adds extra value: it notes that the 'gene' parameter accepts a comma-separated list and that using a list returns a verdict per gene. It also clarifies the default for 'maf' and 'mask'. This goes beyond the schema, improving parameter understanding for batch usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear question defining the tool's purpose: 'Does a gene-trait association replicate across ancestries and biobanks?' This directly states the function. It also distinguishes itself as a specialized view that handles a SQL double-counting issue, which helps set it apart from generic query tools like the sibling 'query'. However, the purpose is embedded within technical reasoning, slightly reducing clarity for a general AI agent.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage advice: passing a comma-separated list to screen multiple genes, noting that one gene at a time costs a call each. It also explains the verdict distinction between 'underpowered' and 'discordant'. However, it does not explicitly say when to use this tool versus its siblings ('query', 'schema', 'variants') or when not to use it. The guidance is present but implicit and incomplete.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, and idempotentHint, so the description's behavioral burden is lower. The description adds value by disclosing the exact size of the table ('61.8 million rows'), the cost profile ('local, so a query costs no network'), and the fact that results are capped (25,000 characters, max rows 500). This goes beyond basic safety disclosure to provide performance expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear introductory sentence, a list of table contents, a callout to call schema() first, and a recommendation to use the results view. Every sentence adds value. It could be slightly more concise by removing the exact row count and focusing on the most critical guidance, but it's not overly verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of the domain (61.8 million rows, multiple dimensions per cell), the rich annotations (readOnly, openWorld, idempotent), a fully described 2-parameter schema, and the presence of an output schema, the description is remarkably complete. It explains the exact contents, potential pitfalls (semantic traps), and best practices. There is no need to discuss return values since an output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides clear descriptions for both parameters (sql and max_rows) with examples and constraints. Schema description coverage is 100%, so the baseline is 3. The description does not add meaning beyond what the schema provides, as the schema already explains the allowed SQL statements, default row count, and character limit. The description's mention of capping (500 rows, 25,000 chars) is redundant with the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs 'read-only SQL over the whole BRaVa gene-level results table.' It specifies the verb ('query'), resource ('BRaVa gene-level results table'), and scope ('read-only SQL'). While the name 'query' is generic, the description distinguishes it from siblings by specifying the exact domain (BRaVa) and table type (gene-level results), making it clear what this tool does compared to 'variant' or 'schema'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises to 'call `schema()` first' to understand the correct tables, columns, and semantic traps. It warns about semantic traps that can 'invert the answer.' It recommends querying the `results` view instead of raw tables. This provides explicit when-to-use guidance and a prerequisite action (calling schema()), which is excellent for avoiding misuse.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• variantsA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds rich behavioral context beyond the annotations: explains the HTTP fetch mechanism, reasons for not storing the data locally (large size, frequent upstream rebuilds), caching behavior (permanent caching after first fetch), and links to external data (gnomAD). This goes far beyond the readOnlyHint, openWorldHint, and idempotentHint annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured into three clear paragraphs, front-loaded with the core purpose. It contains some explanatory detail (e.g., HTTP fetch rationale) that is valuable but slightly verbose. Overall, each sentence contributes to understanding the tool's behavior and usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, the description need not detail return values. It covers purpose, modes, differentiation from siblings, caching, and external references. It does not explicitly mention pagination or error handling, but the schema descriptions for limit/offset and context signals provide that. The description is largely complete for a read-only data retrieval tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, providing baseline 3. The description adds significant value for the 'gene' parameter by explaining it enables additional per-biobank effect-direction and cross-biobank replication evidence. It also reinforces the role of 'max_p' implicitly through mention of the threshold. For other parameters, the schema descriptions suffice.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'single-variant results for a trait, genome-wide or inside one gene', explicitly distinguishing it from gene-level burden tests (likely a sibling tool). It uses a specific verb (returns/fetches) and resource (variants), and explains the two distinct modes of operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use genome-wide vs per-gene mode and contrasts with gene-level burden tests. It implies usage context well but does not explicitly state 'when not to use' or name alternative tools. The guidance on caching and HTTP fetch provides operational context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• schemaA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint, openWorldHint, and idempotentHint, which signal safety and non-mutating behavior. The description adds significant behavioral value beyond these: it details specific pitfalls (e.g., effect sizes mismatched to p-values, overlapping ancestry strata, p-values of exactly zero) that would otherwise be opaque. It also describes the return structure comprehensively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately concise but slightly verbose in the second paragraph (listing all pitfalls could be slightly tighter). However, it front-loads the critical usage instruction ('Read this before querying') and the return list. Every sentence adds value and earns its place by providing context not in structured fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, no nested objects, and a likely output schema (mentioned but not provided here), the description is remarkably complete. It covers purpose, usage timing, return structure, and even specific pitfalls. The contextual signals (parameter count 0, schema coverage 100%) confirm no gaps need filling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters and schema description coverage is 100% (though empty schema). The description adds no parameter details because there are none, but this is appropriate and complete for a parameterless tool. No further explanation needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'tables, columns, recipes, vocabulary, thresholds and pitfalls' – a specific resource (schema/metadata) with a distinct verb 'returns'. It distinguishes itself from siblings like 'query' and 'variants' by describing this as a metadata discovery tool rather than a data retrieval tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Read this before querying,' setting a clear usage context: this should be called before using sibling tools (e.g., 'query') to understand the data structure, pitfalls, and analysis vocabulary. It implies when-not-to-use by framing itself as preparatory, not transactional.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.