Server Quality Checklist

Tool Scores

• executeC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states 'perform the chosen action' which implies a write/mutation operation, but doesn't disclose critical behavioral traits: whether this is synchronous/asynchronous, what permissions are required, whether it's idempotent, what happens on failure, or what the response contains. For a tool that appears to execute potentially complex operations/workflows, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that gets straight to the point without unnecessary words. However, this conciseness comes at the cost of being under-specified for a tool with three required parameters and no annotations - it could benefit from one more clarifying sentence about what gets executed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 required parameters (including a nested object), no annotations, and no output schema, the description is inadequate. It doesn't explain what types of actions can be executed, what the expected outcomes are, or provide any error handling context. The agent would struggle to use this tool effectively without trial and error.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description adds no parameter-specific information beyond the generic 'using the provided details' - it doesn't clarify the relationship between execution_type, uuid, and inputs, or provide examples of valid input objects. Baseline 3 is appropriate when the schema does all the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Perform the chosen action for the user using the provided details' is tautological - it essentially restates the tool name 'execute' without specifying what kind of actions or resources are involved. While it mentions 'chosen action' and 'provided details', it doesn't clarify whether this executes operations, workflows, scripts, or other entities, nor does it distinguish this from sibling tools like 'load_execution_info' or 'search_apis'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance - it mentions 'chosen action' and 'provided details' but gives no explicit when-to-use criteria, no prerequisites, and no comparison to alternatives like 'load_execution_info' (which presumably loads execution information rather than performing execution) or 'search_apis'. The agent must infer usage from the parameter schema alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_execution_infoC
  2.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but offers limited behavioral insight. It mentions retrieving information and saving to a file, but doesn't disclose critical traits like whether this is a read-only operation, if it modifies state, authentication needs, rate limits, or error handling. The description adds some context but leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately concise but not optimally structured. The first sentence clearly states the purpose, but the second sentence adds implementation details for coding agents that may not apply to all users. The information is somewhat front-loaded but could be more focused.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description should provide more complete context. It mentions retrieving details and saving to a file, but doesn't explain what format the details take, what 'jentic.json' contains, or what happens if parameters are invalid. For a tool with 2 required parameters and mutation implications, this is inadequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are well-documented in the schema. The description doesn't add meaningful semantic context beyond what the schema provides about operation_uuids and workflow_uuids. It mentions 'configuration files' but doesn't clarify how parameters relate to this output.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Get more details about a specific action' which indicates its purpose, but it's vague about what 'details' means and doesn't clearly differentiate from sibling tools like 'execute' or 'search_apis'. It mentions saving to 'jentic.json' which adds specificity but confuses the core purpose with implementation details for coding agents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some guidance ('Usually used if the action requires specific inputs') but lacks explicit when-to-use vs. alternatives. It doesn't clarify when to choose this over 'execute' or 'search_apis', and the coding agent note is situational rather than general usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_apisA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the tool searches for actions or information but lacks details on behavioral traits such as response format, error handling, rate limits, or authentication needs. For a search tool with zero annotation coverage, this is a significant gap, though it does imply it returns results (e.g., 'to return' in the schema).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with two sentences that efficiently convey purpose and usage guidelines. Every sentence earns its place by providing essential information without redundancy. It could be slightly more concise by integrating examples more seamlessly, but overall it's well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no output schema, no annotations), the description is adequate but has clear gaps. It covers purpose and usage well but lacks behavioral details like return format or error handling. Without annotations or an output schema, the description should do more to compensate, making it minimally viable but incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description does not add any additional meaning beyond what the schema provides, such as explaining how 'capability_description' interacts with 'keywords' or detailing search algorithms. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search for available actions or information based on what the user wants to do.' It provides a specific verb ('search') and resource ('available actions or information'), with examples like 'find Discord servers' and 'send a message' to illustrate usage. However, it doesn't explicitly distinguish this from sibling tools like 'execute' or 'load_execution_info', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool: 'Use this first to understand what's possible.' This clearly indicates it should be employed as an initial step to discover capabilities before invoking other tools, effectively differentiating it from siblings like 'execute' (which likely performs actions) and 'load_execution_info' (which might retrieve details).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.