Server Quality Checklist

Tool Scores

• oe_history_listA
  4.1/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the tool as readOnly and idempotent. The description adds significant behavioral context: 'No side effects', the privacy-reduced default list, and the privacy exposure risk of include_raw=true. This goes beyond the annotations and is crucial for safe use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, consisting of three sentences that front-load the purpose and usage. Every sentence adds value, though the parameter listing could be integrated more smoothly. No redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description should explain return values. It mentions 'privacy-reduced list' but lacks details on volume, format, or pagination behavior. The tool has four parameters and moderate complexity; missing information on result structure reduces completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain parameters. It merely lists inputs ('limit, offset, optional search, optional include_raw=false') without explaining their meaning, behavior, or constraints. For example, 'search' could filter results, but this is not clarified. The defaults and limits from the schema are not reiterated in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists prior OpenEvidence articles, with a specific verb ('List') and resource ('prior OpenEvidence articles'). It distinguishes from sibling tools like oe_article_get (single article retrieval) and oe_ask (asking questions) by specifying the use case: inspecting prior work or obtaining an article_id.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use: 'when the user asks to inspect prior OpenEvidence work or needs an article_id'. It also provides a critical warning about include_raw: 'must be used only with explicit user intent'. However, it does not explicitly list alternative tools for other scenarios, though the sibling context is implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• oe_article_waitA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnly, idempotent), description adds 'Requires authenticated session', 'No side effects', and warns about include_raw exposing private context. Adds value without contradicting annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences plus a concise list, front-loaded with purpose, no superfluous words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers usage, parameters, side effects, and authentication. Lacks return structure details but acceptable given tool's wait-and-return nature and no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, description lists parameters and explains include_raw risk, but omits details on timeout_sec and poll_interval_ms effects beyond defaults. Adequate but not thorough.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool waits for an article to finish and returns normalized fields, distinguishing it from siblings like oe_ask (initiation) and oe_article_get (fetching).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use after oe_ask with wait_for_completion=false, especially for long requests. Provides clear usage context but no explicit exclusions or alternatives beyond implied chain.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• oe_askA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Openly states the side effect of creating a question/article in the user's account and mentions privacy-reduced data return. This goes beyond annotations which only indicate readOnly=false and idempotent=false, providing concrete behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with 4-5 sentences, each earning its place. It front-loads the purpose and adds practical guidance without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Provides sufficient context for an agent to use the tool correctly, including async pattern and follow-up. Lacks details on return structure and error handling, but overall complete given the complexity of 9 parameters and no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description explains the key parameters original_article_id and wait_for_completion with their usage intent. However, it does not address other important parameters like timeout_sec, personalization_enabled, or article_type, leaving some gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Create an OpenEvidence research question' with a specific verb and resource, and explicitly distinguishes from medical advice. It also implies differentiation from the sibling oe_article_wait by describing the async workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises when to set wait_for_completion=false and use oe_article_wait for long questions, and instructs to use original_article_id only for true follow-up continuity. This provides clear when-to and when-not-to guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• oe_article_getA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint; description reinforces with 'No side effects' and adds details about default return fields and privacy risks of include_raw.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four concise sentences, front-loaded with action and resource, followed by usage context, parameter description, and security note. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, description covers return fields; annotations and schema cover safety. Description adds usage context and parameter semantics, making it complete for this simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description explains both parameters: article_id as UUID, include_raw as optional boolean with default false and a critical caution about private thread exposure.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'Fetch an OpenEvidence article by article_id.' Distinguishes from siblings by specifying it follows history lookup or oe_ask, making it unique among related tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to use (after history or oe_ask returns ID), warns about include_raw requiring explicit user intent, and notes authenticated session needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• oe_auth_statusA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly and idempotent hints. The description adds valuable context: requirement for a local profile, potential failure modes, and return structure (boolean + metadata). This goes beyond the annotations, fully disclosing behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no redundancy. The first sentence states the core purpose, the second provides usage context, and the third covers prerequisites and failure modes. Every sentence is essential and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters or output schema, the description sufficiently covers purpose, return value, prerequisites, side effects (none), and failure scenarios. It is fully adequate for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist and schema coverage is 100%. The description implicitly confirms no inputs are needed. Since no additional parameter info is required, the description adds full clarity for the parameterless interface.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks authentication status and returns a boolean and metadata. It explicitly ties its use to sibling tools (history/article/ask) and when auth state is unknown, distinguishing its role from those tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: use before other tools when auth state is unknown, requires a specific browser profile, and notes failure conditions (missing/expired/network). Also states no side effects, helping the agent decide when to call it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.