Server Quality Checklist

Tool Scores

• researchTopicB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only mentions using Exa.ai's capabilities and returning 'current information and practical implementation guidance'. It does not disclose behavioral traits like rate limits, result format, or any constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 20 words, concise and front-loaded with the core purpose. It avoids fluff but could include usage guidance without much increase in length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description is minimally adequate. However, it lacks differentiation from sibling tools and does not address behavioral aspects, leaving gaps for an AI agent to infer.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameter is already well-documented in the schema. The tool description adds no additional meaning or context beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('conduct'), resource ('web research using Exa.ai'), and scope ('software development topics'). It effectively distinguishes from sibling tools like deepResearch by specifying 'quick, focused' research.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives like deepResearch or askFollowUp. The description does not specify when not to use it or provide context for choosing among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deepResearchB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does not mention any behavioral traits such as rate limits, costs, internal processes, or side effects. For a research tool, the description lacks important context about what happens during the research or how results are returned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently conveys the tool's purpose and target scenarios. No unnecessary words or repetition, earning high marks for conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one required parameter and no output schema or annotations, the description provides adequate context about purpose but lacks behavioral details. It is sufficiently complete for a simple tool, but could be improved with transparency about the research process or limitations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema's parameter 'topic' has a detailed description that already covers semantic guidance, including recommendations for technical requirements, architectural considerations, etc. The tool description does not add further information about parameters beyond what the schema provides, so the baseline score of 3 is appropriate given 100% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool conducts comprehensive, in-depth research using Exa.ai for critical decision-making and complex architectural planning. It specifies the verb 'research' and the resource 'Exa.ai's exhaustive analysis capabilities,' but does not explicitly differentiate it from the sibling tool 'researchTopic', which likely has a similar purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for high-stakes situations like critical decision-making and architectural planning, but it does not provide explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives among sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runAndExtractB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must fully disclose behavior. Mentions security controls and extraction, but doesn't explain output format, error handling, or limitations of the extraction process.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no redundant information, purpose is front-loaded and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Three required parameters, no output schema, no annotations. Description does not explain what the tool returns or how extraction results are structured, leaving significant gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for each parameter. The description adds little beyond restating purpose; no new parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states specific verb 'Execute terminal commands' and resource 'output extraction', distinguishing it from sibling tools which are about asking questions or research.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage through security controls and cross-platform support, but lacks explicit when-to-use, when-not-to-use, or alternative tool references.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• askAboutFileA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description conveys read-only nature but lacks details on error handling for unsupported files, authorization needs, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence that front-loads the core purpose and covers key aspects without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Basic but sufficient given the tool's simplicity. Missing details on return format or error cases, but not critical for a straightforward extraction tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions. The tool description adds minimal extra meaning beyond listing supported file types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it extracts specific information from files without loading entire content, lists supported file types, and distinguishes from reading full files.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when needing specific info from files without full context, but does not explicitly state when not to use or mention alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• askFollowUpA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility. It mentions the prerequisite (after runAndExtract) but does not disclose what happens if called without that prerequisite or any other behavioral traits like error handling or idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences, no redundant words. It front-loads the core purpose and adds a critical usage constraint in the second sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema, no annotations), the description is mostly adequate. However, it lacks information about the return value or error behavior, which could hinder an AI agent understanding edge cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% coverage with one required parameter 'question' described as 'Follow-up question about the previous terminal command execution and its output.' The description adds meaningful context beyond the schema by specifying the scope of the question.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Ask follow-up questions about the previous terminal command execution without re-running the command.' It specifies the verb (ask), resource (previous terminal command execution), and distinguishes it from re-running the command.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'Only available after using runAndExtract tool.' It also clarifies what it avoids: 'without re-running the command.' This provides good usage context, though it does not explicitly mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.