Server Quality Checklist

Tool Scores

• get_emailsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the sorting behavior, which is useful, but lacks critical details such as pagination, rate limits, authentication needs, error conditions, or what 'all emails' entails (e.g., time range, limit). For a read operation with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It front-loads the core purpose and includes essential behavioral detail (sorting), making it appropriately sized and well-structured for its purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete. It lacks information on return values (e.g., email format, fields), error handling, and operational constraints. For a tool with one parameter but unknown output complexity, this leaves significant gaps for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'email' documented as 'The inbox email address'. The description adds no additional meaning beyond this, as it doesn't clarify parameter usage or constraints. Baseline 3 is appropriate since the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List all emails') and resource ('received in an inbox'), with specific sorting behavior ('sorted by date, newest first'). It distinguishes from siblings like 'wait_for_email' by focusing on listing rather than waiting, but doesn't explicitly differentiate from potential overlapping functions with other tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'wait_for_email' or 'list_inboxes'. The description implies usage for retrieving emails from a specific inbox but doesn't mention prerequisites, exclusions, or comparative contexts with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_quotaB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions checking API usage and quota, which implies a read-only operation, but doesn't specify whether this requires authentication, has rate limits, returns real-time or cached data, or what format the output takes. This leaves significant gaps for a tool that interacts with API limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without any wasted words. It's front-loaded with the core purpose, making it easy to parse and understand immediately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate as a basic overview. However, for a quota-checking tool, it lacks details on output format, authentication needs, or potential side effects, which could be important for an AI agent to use it correctly in various contexts.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the schema description coverage is 100%, so there's no need for parameter details in the description. The description appropriately focuses on the tool's purpose without redundant parameter information, earning a baseline score of 4 for zero-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Check') and resource ('API usage and remaining monthly quota'), making it immediately understandable. However, it doesn't differentiate from potential sibling tools that might also retrieve quota information, though none are listed among the provided siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, prerequisites, or specific contexts. It simply states what the tool does without indicating scenarios where it's appropriate or inappropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_inboxesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It implies a read-only operation by using 'List', but doesn't specify whether this requires authentication, has rate limits, returns paginated results, or what 'active' means (e.g., time-based criteria). For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It's front-loaded with the core action and resource, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate as a basic read operation. However, without annotations or output schema, it lacks details on return format (e.g., list structure, fields like inbox IDs or statuses) and behavioral context, leaving room for improvement in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (though empty). The description doesn't need to add parameter information, so it meets the baseline expectation. No additional semantic value is required or provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and resource ('all your active temporary email inboxes'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_quota' which might also involve inbox listing aspects, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'get_emails' or 'wait_for_email', nor does it mention any prerequisites or exclusions. It simply states what the tool does without context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_inboxB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses the destructive nature ('Delete... and all its emails'), which is critical, but lacks details on permissions, reversibility, error conditions, or rate limits. For a mutation tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action and resource. Every word earns its place, with no redundant or vague phrasing, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's destructive nature, no annotations, and no output schema, the description is incomplete. It lacks information on success/error responses, side effects beyond deletion, or operational constraints, which are essential for safe and effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'email' fully documented in the schema. The description does not add meaning beyond the schema (e.g., no format examples or constraints), so it meets the baseline of 3 where the schema handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Delete') and resource ('a temporary email inbox and all its emails'), distinguishing it from siblings like create_inbox (creation) or list_inboxes (listing). It precisely defines the scope of deletion, including associated emails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for deleting temporary inboxes, but does not explicitly state when to use this tool versus alternatives (e.g., no guidance on prerequisites like needing an existing inbox or comparisons to other deletion-related tools). It lacks explicit exclusions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wait_for_emailA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the blocking behavior and timeout mechanism, which are key traits. However, it lacks details on permissions, rate limits, error handling, or what happens after timeout (e.g., returns null or error), leaving gaps for a tool with potential real-time implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence and adds essential behavioral detail in the second. Both sentences earn their place by providing critical information without redundancy, making it highly efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (blocking wait with timeout), no annotations, and no output schema, the description is partially complete. It covers the basic operation but omits details like return values (e.g., email content or timeout status), error conditions, or dependencies on other tools (e.g., requires an existing inbox from 'create_inbox'), leaving room for improvement.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('email' as inbox address and 'timeout' with range/default). The description adds no additional meaning beyond what the schema provides, such as clarifying parameter interactions or usage nuances, resulting in a baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('wait for a new email to arrive'), the resource ('inbox'), and distinguishes it from siblings like 'get_emails' (which likely retrieves existing emails) by emphasizing the blocking/waiting behavior. It precisely communicates the tool's function without being tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying it blocks until an email arrives or timeout is reached, suggesting it's for real-time monitoring rather than historical retrieval. However, it doesn't explicitly state when to use this versus alternatives like 'get_emails' or 'wait_for_otp', nor does it provide exclusions or prerequisites, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_inboxA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: the inbox is 'temporary disposable' and returns an address that 'can receive emails.' However, it lacks details on permissions, rate limits, or what happens upon expiration (beyond TTL parameter). It doesn't contradict annotations (none exist).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and followed by usage guidelines. Every sentence adds value: the first defines the tool, and the second provides critical context for when to use it. Zero waste or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and 3 parameters with full schema coverage, the description is mostly complete. It covers purpose and usage well but lacks behavioral details like response format or error handling. For a creation tool with no output schema, it could benefit from mentioning return value specifics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all three parameters. The description adds no parameter-specific information beyond what's in the schema, but it implies the tool's purpose relates to the parameters (e.g., TTL for expiration). Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Create a temporary disposable email inbox') and the resource ('email address that can receive emails'). It distinguishes from siblings like 'delete_inbox' or 'get_emails' by focusing on creation rather than deletion or retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool: 'Use this when you need to sign up for a service, verify an account, or receive a one-time code.' It provides clear context and distinguishes from alternatives like 'get_otp' or 'wait_for_email' by focusing on inbox creation for initial setup.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_otpA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the core behavior (extracting OTP from the latest email) and includes a fallback mechanism (include_body for raw text if OTP not found), which adds useful context. However, it lacks details on error handling, rate limits, or authentication needs, leaving some behavioral aspects unclear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences that are front-loaded with the core purpose and usage guidelines. Every word earns its place, with no redundancy or unnecessary elaboration, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is mostly complete. It covers purpose, usage, and some behavioral aspects, but lacks details on output format, error cases, or dependencies on other tools like wait_for_email. This leaves minor gaps in full contextual understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents both parameters. The description adds no additional parameter semantics beyond what the schema provides, such as clarifying email format or OTP extraction logic. However, with only 2 parameters and high schema coverage, the baseline is 3, and the description's clarity on tool purpose slightly elevates this to 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Extract OTP/verification code') and resource ('from the latest email in an inbox'), distinguishing it from siblings like get_emails (which retrieves emails generally) or wait_for_otp (which waits for OTP arrival). It precisely defines the tool's function without redundancy.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool ('when an email has already arrived and you need to extract the code'), providing clear context. It implicitly distinguishes from alternatives like wait_for_otp (which waits for arrival) and get_emails (which retrieves emails without OTP extraction), though it doesn't name them directly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wait_for_otpA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and effectively discloses key behavioral traits: it blocks execution until an email arrives or timeout occurs, extracts codes automatically, and specifies a timeout range (implied by schema). It doesn't detail error handling or exact extraction logic, but covers core behavior well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste: the first states the core functionality and behavior, the second provides usage context. It's front-loaded with essential information and appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is mostly complete for this tool's complexity—it explains blocking, automation, and use case. However, it lacks details on output format (e.g., what the extracted code looks like) and potential failure modes, leaving minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters fully. The description adds no additional parameter semantics beyond what the schema provides, such as format examples or edge cases, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('wait for', 'extract') and resources ('email', 'OTP/verification code'), distinguishing it from siblings like get_otp or wait_for_email by emphasizing automation and blocking behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use this tool ('most useful tool for account verification flows') and implies alternatives by mentioning siblings like get_otp (which likely doesn't block) or wait_for_email (which might not extract codes), providing clear context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.