Server Quality Checklist

Tool Scores

• list_jobsD
  1.8/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure but offers none. 'List jobs' doesn't indicate whether this is a read-only operation, whether it requires authentication, what format results are returned in, if there are rate limits, or any other behavioral characteristics. The description fails to provide any operational context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  While technically concise with just two words, this description represents under-specification rather than effective brevity. The single phrase 'List jobs' fails to provide necessary context that would help an agent understand when and how to use this tool. Conciseness should not come at the expense of clarity and completeness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, the description 'List jobs' is completely inadequate. It doesn't explain what constitutes a 'job' in this context, what information is returned, whether results are paginated, or any other operational details. The description fails to provide the minimal context needed for effective tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the input schema has 100% description coverage (though empty). With no parameters to document, the description doesn't need to compensate for schema gaps. The baseline for zero-parameter tools is 4, as there's no parameter semantics burden on the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'List jobs' is a tautology that merely restates the tool name without adding meaningful context. It specifies the verb ('list') and resource ('jobs'), but provides no differentiation from sibling tools like 'get_job_status' or additional scope information. This minimal description fails to clarify what type of jobs are being listed or under what conditions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides absolutely no guidance on when to use this tool versus alternatives like 'get_job_status' or other sibling tools. There's no mention of appropriate contexts, prerequisites, or exclusions. Users must infer usage patterns from the tool name alone, which is insufficient for effective tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_email_sendC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'verify an email using trykitt' but fails to explain key traits like whether this is a read-only or mutative operation, what the expected outcome is (e.g., sends an email, returns a status), or any rate limits or authentication needs. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded with the main purpose, followed by parameter details in a structured 'Args:' section. It avoids unnecessary elaboration, but the lack of context and behavioral details means it could be more informative without sacrificing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (2 parameters, no annotations, no output schema), the description is incomplete. It doesn't explain what 'verify' means in practice, what happens after invocation (e.g., sends an email, returns a job ID), or how it relates to sibling tools, leaving the agent with insufficient context for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description lists parameters ('email' and 'custom_data') and notes that 'custom_data' is optional, adding basic semantics beyond the input schema. However, with 0% schema description coverage, it doesn't fully compensate by explaining parameter formats (e.g., email validation rules, custom_data structure), leaving the agent with incomplete information for proper usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'verify an email using trykitt' which provides a basic verb+resource combination, but it's vague about what verification entails (e.g., sending a verification email, checking validity). It doesn't distinguish from siblings like 'find_email' or 'get_job_status', leaving ambiguity about the specific action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'find_email' or 'get_job_status'. The description lacks context about prerequisites, such as whether this initiates a verification process or checks an existing one, leaving the agent without clear usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_emailC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'finds' an email address, implying a read-only operation, but does not specify accuracy, data sources, rate limits, or authentication needs. For a tool with no annotations and potential privacy implications, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the purpose, followed by parameter details. It uses a clear structure with bullet points for args. However, the parameter explanations are very brief and could be more informative, slightly reducing efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of finding email addresses, no annotations, no output schema, and low parameter coverage, the description is incomplete. It lacks details on return values, error handling, data sources, and accuracy, which are crucial for effective tool use. The description does not adequately compensate for the missing structured data.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal semantics beyond the input schema. It lists parameters with brief explanations (e.g., 'full_name: The full name of the person'), but with 0% schema description coverage, it does not fully compensate. The explanations are basic and do not provide format details or usage examples, leaving gaps for the required parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find an email address for a person.' It specifies the verb ('find') and resource ('email address'), but does not distinguish it from sibling tools like 'verify_email_send', which might have overlapping functionality. The purpose is specific but lacks sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention sibling tools like 'verify_email_send' or specify contexts where this tool is preferred. Usage is implied only through the parameter descriptions, but no explicit when/when-not instructions are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_statusB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'Get[s] the status of a job,' implying a read-only operation, but doesn't clarify aspects like whether it requires authentication, has rate limits, returns specific status formats (e.g., pending, completed), or handles errors. This leaves significant gaps for an agent to understand how to use it effectively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and well-structured. It starts with a clear purpose statement, followed by a brief 'Args' section that lists the parameter with a simple explanation. There's no unnecessary information, and every sentence serves a functional role in guiding usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't cover behavioral aspects like authentication needs, error handling, or what the status output looks like (e.g., string values, timestamps). For a tool that likely returns critical operational data, this leaves the agent without enough context to use it reliably in complex scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context for the single parameter: 'job_id: The ID of the job to check.' This clarifies that 'job_id' is an identifier used to retrieve status, which is helpful since schema description coverage is 0% (the schema only provides a title and type without explanation). With one parameter, the baseline is 4, and the description compensates well by explaining its purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get the status of a job.' It uses a specific verb ('Get') and resource ('status of a job'), making the function unambiguous. However, it doesn't differentiate from sibling tools like 'list_jobs', which might provide a broader overview rather than specific status checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'list_jobs' for listing multiple jobs or other tools for related operations. There's no context about prerequisites, such as needing a job ID from another operation, or when this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.