Server Quality Checklist

Tool Scores

• list_filtersA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool returns a CSV-formatted table, which is useful behavioral context beyond just listing. However, it doesn't cover other aspects like rate limits, error handling, or authentication needs, leaving gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the purpose clearly. The subsequent sentences add necessary context, parameter explanations, return format, and an example without redundancy. Every sentence earns its place, though it could be slightly more streamlined.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters, no annotations, and no output schema, the description is moderately complete. It covers purpose, parameters with examples, and return format, but lacks details on behavioral traits like error cases or performance. For a simple list tool, this is adequate but has clear gaps in transparency.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It adds meaning by explaining that 'mart' is an identifier (e.g., 'ENSEMBL_MART_ENSEMBL') and 'dataset' is an identifier (e.g., 'hsapiens_gene_ensembl'), and provides an example with specific values. This clarifies the parameters beyond the schema's basic titles, though it doesn't detail all possible values or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Lists all available filters for a given dataset' and 'returns all filters that can be applied to the specified dataset.' It specifies the verb ('lists'), resource ('filters'), and scope ('for a given dataset'), but doesn't explicitly differentiate it from sibling tools like list_datasets or list_marts, which is why it's not a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some context: 'Filters are used to narrow down the results of a Biomart query' and 'This function returns all filters that can be applied to the specified dataset,' which implies usage in query preparation. However, it lacks explicit guidance on when to use this tool versus alternatives like list_common_attributes or list_all_attributes, and doesn't mention prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dataA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and adds valuable behavioral context beyond basic functionality. It discloses 'automatic retry logic for resilience' which is important operational behavior, and specifies the return format ('CSV-formatted results'). However, it doesn't mention potential rate limits, error conditions, or performance characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with purpose statement, functional details, parameter documentation, return specification, and example. While slightly longer than minimal, every section adds value. The front-loaded purpose statement is clear, and the example is particularly helpful. Minor deduction for some redundancy between the opening statement and the detailed explanation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters with 0% schema coverage and no output schema, the description provides substantial context: clear purpose, parameter semantics with examples, return format specification, and behavioral details. The main gap is lack of explicit error handling guidance or performance expectations, but for a query tool with good parameter documentation, this is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by providing detailed parameter semantics. Each of the 4 parameters is clearly explained with examples: mart ('mart identifier'), dataset ('dataset identifier'), attributes ('list of attributes to retrieve'), filters ('dictionary of filters to apply'). The examples make the parameter usage concrete and understandable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('queries', 'performs data retrieval') and resources ('Biomart', 'biological data'). It distinguishes this tool from siblings by specifying it's for main data retrieval with attributes and filters, unlike list_* tools that only list metadata or translation tools that convert data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('main data retrieval from Biomart'), but doesn't explicitly state when NOT to use it or name specific alternatives. It implies usage for querying actual data rather than listing metadata (like list_datasets), but doesn't provide explicit exclusions or comparison to siblings like get_translation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_martsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: it returns a CSV string (not a structured object), provides the exact format of the output including column names, and includes a concrete example showing the structure of returned data. It doesn't mention performance characteristics like rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured: first sentence states the purpose, second explains the context/hierarchy, third describes the return format, and fourth provides a concrete example. Every sentence adds value with zero wasted words, and the information is appropriately front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless tool with no annotations and no output schema, the description provides excellent coverage: purpose, context, return format, and example. The only minor gap is the lack of explicit error handling or edge case information, but for this simple tool, the description is nearly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema coverage, the baseline would be 4. The description appropriately notes there are no parameters by showing 'list_marts()' in the example and doesn't attempt to add unnecessary parameter information, which is correct for a parameterless tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Lists all available Biomart marts') and resource ('from Ensembl'), distinguishing it from siblings like list_datasets or list_filters by focusing on the mart level of the Biomart hierarchy. It provides a concrete example of what the tool returns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly provides usage context by explaining the Biomart hierarchy (MART -> DATASET -> ATTRIBUTES/FILTERS), suggesting this tool should be used first to identify marts before accessing datasets or attributes. However, it doesn't explicitly state when to use alternatives like list_datasets or list_filters.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_translationA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behaviors: it's a translation operation with caching for performance, returns a string result or error message, and provides a concrete example of the expected output format. It doesn't mention rate limits, authentication needs, or error handling specifics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized: purpose statement, behavioral context, parameter documentation, return value explanation, and a concrete example. Every sentence adds value with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 5-parameter tool with no annotations and no output schema, the description provides excellent parameter documentation and behavioral context. The main gap is the lack of explicit error handling details beyond 'error message if not found' - more specifics about error formats or conditions would help.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by providing detailed parameter explanations with clear examples for all 5 parameters. Each parameter's purpose and example values are clearly documented, adding significant value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Translates a single identifier from one attribute type to another' with specific examples like converting gene symbols to Ensembl IDs. It distinguishes from sibling 'batch_translate' by specifying 'single identifier' translation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (single identifier translation) and implies when not to use it (batch operations should use 'batch_translate'). However, it doesn't explicitly name alternatives or provide exclusion criteria beyond the single vs. batch distinction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_all_attributesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by disclosing key behavioral traits: it warns that the function 'can return a large number of attributes and may be unstable for certain datasets', specifies that it filters out 'homologs, microarray probes', and notes the output format as CSV. This covers performance, stability, and output behavior beyond basic functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded: the first sentence states the purpose, followed by details on filtering, cautions, and usage advice. Each sentence adds value without redundancy, and it includes a helpful example at the end.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters, no annotations, and no output schema, the description is quite complete: it explains purpose, parameters, behavior, cautions, and alternatives. However, it doesn't detail error conditions or exact filtering criteria beyond examples, leaving minor gaps for a tool with potential instability.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate fully. It does so by explaining both parameters: 'mart' as 'The mart identifier' with an example, and 'dataset' as 'The dataset identifier' with an example. This adds essential meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'lists all available attributes for a given dataset with some filtering' and specifies it returns a filtered list. It distinguishes from the sibling 'list_common_attributes' by mentioning that tool as an alternative. However, it doesn't fully differentiate from other siblings like 'list_filters' or 'list_datasets' in terms of resource scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: it advises to 'consider using list_common_attributes first' due to potential instability and large response sizes, and it mentions filtering out less commonly used attributes. This gives clear context on when to use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_datasetsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses key behavioral traits: it returns all datasets (no filtering), outputs as a CSV string, and includes an example showing the return format. It does not mention performance aspects like rate limits or error handling, but covers the core behavior adequately for a read-only operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded: the first sentence states the purpose, followed by explanatory context, parameter details, return value, and an example. Every sentence adds value without redundancy, and it is appropriately sized for a single-parameter tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no output schema, no annotations), the description is nearly complete: it explains purpose, usage, parameters, and return format with an example. A minor gap is the lack of explicit error handling or edge cases, but it suffices for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It fully documents the single parameter 'mart' by explaining its role ('mart identifier to list datasets from'), providing valid values (e.g., ENSEMBL_MART_ENSEMBL), and including an example. This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Lists') and resource ('all available biomart datasets for a given mart'), distinguishing it from siblings like list_marts (which lists marts) and list_filters (which lists filters). It specifies the scope ('all datasets available in the specified mart') and output format ('CSV string'), making the purpose highly specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by explaining that 'Each mart contains multiple datasets' and this tool lists them, implying it should be used after selecting a mart. However, it does not explicitly state when to use alternatives like list_marts (for listing marts) or get_data (for fetching dataset contents), nor does it mention prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_common_attributesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by disclosing key behavioral traits: it explains the tool returns only the most frequently used attributes (defined in COMMON_ATTRIBUTES), specifies the output format (CSV-formatted table), and includes an example. It doesn't mention rate limits or auth needs, but covers core behavior adequately.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose, followed by usage guidelines, parameter details, return format, and an example. Every sentence adds value without redundancy, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is largely complete: it covers purpose, usage, parameters, return format, and includes an example. It could mention error handling or data sources, but for a read-only list tool, it provides sufficient context for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It adds meaningful semantics by explaining what 'mart' and 'dataset' represent (identifiers), providing examples (e.g., 'ENSEMBL_MART_ENSEMBL', 'hsapiens_gene_ensembl'), and clarifying they are required. This goes well beyond the basic schema, though it could detail constraints or sources further.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Lists commonly used attributes') and resources ('for a given dataset'), and explicitly distinguishes it from its sibling 'list_all_attributes' by noting it returns only the most frequently used attributes to avoid overwhelming the model.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool versus alternatives: it specifies to use this tool 'to avoid overwhelming the model with too many options' and directs users to 'use list_all_attributes' for a complete list, clearly differentiating from the sibling tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• batch_translateA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It effectively describes the tool's behavior: it's a batch translation operation that returns translations, not-found items, and counts. However, it lacks details on error handling, rate limits, or authentication needs, which would be valuable for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It starts with a clear purpose statement, follows with usage guidelines, details parameters with examples, specifies return values, and provides a practical example. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (5 parameters, no annotations, no output schema), the description is highly complete. It covers purpose, usage, parameter semantics, return structure, and includes an example. This provides all necessary context for an agent to understand and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides detailed parameter semantics in the 'Args' section, including names, types, descriptions, and examples for all 5 parameters (e.g., 'mart (str): The mart identifier (e.g., "ENSEMBL_MART_ENSEMBL")'). This fully explains parameter meanings beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Translates multiple identifiers in a single batch operation.' It specifies the verb ('translates'), resource ('identifiers'), and scope ('batch operation'), and distinguishes it from sibling 'get_translation' by emphasizing efficiency for many identifiers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides usage guidance: 'This function is more efficient than multiple calls to get_translation when you need to translate many identifiers at once.' It names the alternative tool ('get_translation') and specifies when to use this tool ('when you need to translate many identifiers at once').

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.