Server Quality Checklist

Tool Scores

• get_pathway_eventsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description includes ordering by hasEvent_rank and return format, but lacks explicit safety/read-only declaration or truncation details; no annotations to supplement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: purpose, Args, Returns. Front-loaded, no redundant text, efficient for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, parameter, and return structure (including truncation hint). Missing explanation of truncation behavior and any access limitations, but adequate for single-param tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema coverage, the description adds full parameter meaning: stable ID format example and numeric DB_ID, compensating for schema omission.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists ordered child events (sub-pathways and reactions) of a pathway, distinguishing it from siblings like get_object.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives; no mention of when not to use it or context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_objectA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description explains key behaviors: resolving IDs, merging subclass attributes, leaf class wins on clashes, and error response. Lacks mention of authorization or rate limits but is otherwise thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with a brief overview, then Args and Returns sections. Information-dense without being verbose, though could be slightly more compact.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given simple single-parameter tool with no output schema, the description covers input format, return structure, and error handling. Sufficient for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has only type string with no description; the description adds critical detail: id can be a stable ID like 'R-HSA-69278' or numeric DB_ID, and the return example reinforces usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Fetch a full Reactome object' and explains the inheritance chain assembly, distinguishing it from siblings like search_by_name or describe_table.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context for when to use the tool (fetching by stable ID or DB_ID) and gives an example format. Does not explicitly list when not to use or mention alternatives, but context from sibling names implies their use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_participantsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden. It discloses the input type (stable id or DB_ID), the return format including nested structures, and edge-case behavior (empty lists for non-reactions). This is thorough but could explicitly state read-only nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, using a clear structure: purpose statement, Args, Returns. Every sentence adds value without redundancy. It is front-loaded with the main action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having only one parameter and no output schema, the description fully explains the return value structure, including nested dicts for inputs/outputs/catalysts and DB_ID fields. It also covers edge cases, making the tool's behavior completely clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds crucial context: 'A ReactionlikeEvent (Reaction/BlackBoxEvent/...) stable id or DB_ID.' This clarifies what the single 'id' parameter represents, going beyond the schema's generic string type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Get the inputs, outputs, and catalysts of a reaction-like event,' clearly specifying the action and resource. It distinguishes itself from sibling tools like get_object or get_pathway_events by focusing on participants of reactions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving reaction participants but does not explicitly compare with sibling tools or state conditions to avoid using this tool. No when-not guidelines are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_referrersA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully carries the burden. It discloses that the tool is best-effort and not exhaustive, explaining it checks only high-value relations among 132 link tables. This sets clear expectations about the tool's behavior and limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with an intro, then clear arg and return sections. It is informative without being excessively long. Minor redundancy (e.g., mention of 132 link tables could be condensed) but overall efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description provides a detailed return structure with fields. It covers purpose, parameters, return format, and limitations, making the tool fully usable without additional documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds thorough explanations for all three parameters: id (stable id or DB_ID), attribute (optional restriction with examples), and limit (range 1-500, default 50). This provides meaning far beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool does a reverse lookup to find objects pointing at a given one, distinguishing it from REST's awkward inverse navigation. It specifies that it checks a curated set of high-value relations, which differentiates it from sibling tools like get_object or get_participants.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context on when to use the tool—when a reverse lookup is needed—and notes its best-effort, non-exhaustive nature. However, it does not explicitly state when not to use it or mention specific alternatives among siblings, though the uniqueness of the purpose mitigates this.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_tableA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes return format comprehensively. Lacks details on error handling (e.g., table not found) but covers the core behavior well. No annotations, so description carries full burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short paragraphs: purpose first, then args/returns. Every sentence is useful; no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Single parameter adequately explained, return format specified, and reference to sibling tool. Complete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has no description for the 'table' parameter, but the description provides crucial guidance: exact name from list_tables, with examples. This compensates fully for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it shows columns of a table with specific fields (name, type, nullability, key, default, extra). The verb 'show' and resource 'columns' are specific and distinguish it from siblings like list_tables.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Points to list_tables to get exact table name, implying the correct workflow. Does not explicitly state when not to use, but context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavioral traits: writes are impossible (SELECT-only user), auto-LIMIT injection, max_rows with default and truncation, error handling, and return format. All critical behaviors are covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening, bullet-like details, an example, and return format. It is slightly lengthy but every sentence adds value. Front-loads purpose and key restrictions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is remarkably complete. It covers input, output (with fields), restrictions (read-only, auto-LIMIT, error), and even guides the agent to first call schema_overview. No gaps remain for effective tool selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description compensates fully. It explains the sql parameter with constraints (single statement, parameter-free, literals quoted) and provides an example. It also defines max_rows (range 1-5000, default 200) and mentions truncation of long text cells.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Run a single READ-ONLY SQL statement against the Reactome DB.' It specifies accepted statement types (SELECT/WITH/SHOW/DESCRIBE/EXPLAIN) and distinguishes from siblings by recommending schema_overview first.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit context: it is for read-only queries, accepts only one statement, and suggests calling schema_overview first. It gives an example and mentions rejection of multiple statements, but does not explicitly list when not to use this tool versus alternatives like get_object or search_by_name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• schema_overviewA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries full burden. It describes the return content comprehensively (primer on class-table-inheritance, useful tables, join recipes, data release). For a read-only explanation tool, this is transparent; no destructive behavior is expected.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) and well-structured: purpose first, then contents, then usage directive. Every sentence adds value without verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters or output schema, the description is self-contained. It fully explains what the tool returns and mentions the data release loaded and the alternative resource. No gaps for an explanation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters; baseline is 4. The description adds value by detailing what the tool explains, but no parameter documentation is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: explaining the Reactome GKB schema to enable writing correct SQL. It uses specific verbs ('explain') and resources ('schema'), and distinguishes from siblings like describe_table and query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'Read this once before writing `query()` SQL.' This tells the agent when to use the tool and implies it as a prerequisite for querying. It also mentions an alternative source (schema://reactome resource).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_tablesA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses row counts are exact (MyISAM), return structure (dict with count, tables, truncated), and ordering. No annotations provided, so description fully covers behavioral aspects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise with clear Args/Returns sections, examples, and key details. No unnecessary text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for a simple list tool with 2 params. No output schema but return format is explained. Sibling tools cover other needs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage, but the description adds full meaning: filter is case-insensitive substring, limit is max count with default. Includes practical filter examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists tables with exact row counts and engine, specifying the database is MyISAM. It distinguishes from siblings like describe_table (single table) and schema_overview.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides filter examples and limit context (default 500, DB has 242), guiding effective use. Lacks explicit when-not-to-use but implied by sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_by_nameA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description fully discloses behavior: it performs a search on a 1.86M-row supertable, describes the two different query modes (prefix vs substring) and their performance, and details the return structure. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a brief intro, a clear bulleted list of arguments with explanations, and a return format. Every sentence adds value; no redundant or missing information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of 4 parameters and no output schema, the description is fully complete. It covers argument details, performance, return format, and gives practical advice. The return dict structure is explicitly listed, compensating for the missing output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates by explaining all four parameters (query, classes, limit, anchored) with their default values, constraints, and behavioral implications. This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches objects by `_displayName` with a clear verb and resource. It distinguishes from siblings like get_object (which retrieves by ID) and get_participants (which retrieves by specific relationships).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use the anchored parameter, performance implications, and recommendations for hot paths. It also notes the absence of FULLTEXT indexes, helping the agent decide between anchored=True vs anchored=False and using classes filter.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.