Server Quality Checklist

Tool Scores

• cancel_taskC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for disclosing behavioral traits. It does not mention side effects, reversibility, permission requirements, or error behavior for non-running tasks. The description is too terse to be transparent about the operation's impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, which is efficient for such a simple tool. However, it barely adds value beyond the tool name and lacks any additional contextual structure, so it does not fully earn its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, the description does not explain expected outcomes (e.g., success/failure responses, whether cancellation is immediate or asynchronous). Given the tool's mutation-like nature and the existence of siblings, the description is incomplete for safe and correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter meaning, but it does not mention task_id at all. While 'task_id' is somewhat self-explanatory, the description fails to clarify what values are valid or what happens with an invalid ID.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Cancel a running task' uses a specific verb (cancel) and resource (running task), clearly distinguishing it from siblings like run_task or check_task. It is unambiguous and directly states the tool's function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, no prerequisites (e.g., task must be running), and no exclusions. It simply states what it does, leaving the agent without context for appropriate invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reply_taskA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the full burden of behavioral disclosure. It only states the return format is the same as run_task (waiting or done), but does not disclose side effects such as whether the reply mutates task state, whether replies can be sent multiple times, or what happens if the task is not waiting. 'Send a reply' implies an action, but the consequences are not explained.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action and trigger, and contains no filler. Every sentence contributes necessary context: when to call it and what to expect as a return.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no annotations and 0% parameter coverage, the tool is simple (two required scalar params) and has an output schema, so the description does not need to explain return values in depth. It mentions the return format and trigger, which is enough for basic invocation. It falls short only in behavioral and parameter detail, which are already penalized separately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter guidance. It only implies 'message' is the reply content, and does not clarify the role of task_id (e.g., that it comes from run_task) or describe valid message formats. The parameter names are self-explanatory to a degree, but the description adds little semantic value beyond those names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Send a reply') and a specific resource/target (HermitAgent) with an explicit trigger condition ('when run_task returned {status:"waiting"}'). This distinguishes it from siblings like run_task, check_task, and cancel_task by defining its role as the follow-up action for waiting tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear context for use ('when run_task returned {status:"waiting"}') and mentions the return format consistency with run_task. It does not explicitly name alternatives or exclusions, but the trigger condition is specific enough to guide selection among sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_taskA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses asynchronous execution, immediate return of a task_id, and four distinct status states including the interactive 'waiting' state and the 'needs_review' outcome. However, it omits error handling, timeout behavior, cancellation semantics, and whether the task modifies the filesystem, which are relevant for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core action and async behavior, and the return-state list is compact and scannable. There is minor redundancy between the first sentence's 'Immediately returns {status:'running', task_id}' and the subsequent enumerated return values, but overall the structure is effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The state machine and output values are well documented, which covers a major part of the tool's complexity. Still, with 5 parameters, 0% schema coverage, and no annotations, the absence of parameter semantics and failure/error handling leaves significant gaps for an agent attempting to invoke the tool correctly without relying on external documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds only minimal parameter meaning: 'task' is a coding task and the model defaults to qwen3-coder:30b. The required cwd parameter and the optional strategy and max_turns parameters are not explained, so the description does not compensate for the absent schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Run a coding task using a local LLM.' It also names the default model and immediately distinguishes itself from sibling tools by directing the agent to poll with check_task(task_id) and reply via reply_task(task_id, message). This makes it unambiguous that run_task is the initiating tool in the workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool versus its siblings: it launches a task, then the agent should poll with check_task and reply with reply_task when a question is waiting. It names alternatives explicitly, though it does not explicitly say 'use check_task instead of run_task for status checks.' The workflow guidance is strong but not exhaustive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_taskA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so thoroughly. It discloses every possible return status, including nuanced ones like 'needs_review' and 'not_found', and explains the behavior of full=true for retrieving untruncated results.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a one-sentence usage note, a bullet-like list of statuses, and a final parameter tip. Every sentence earns its place with no repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with two parameters and an output schema, the description covers all essential behavior: when to call it, all possible return values, and the meaning of the optional flag. Nothing needed to invoke it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It clearly explains the effect of full=true, and task_id is given meaning through the not_found status. It could additionally state that task_id comes from run_task's response, but the usage line makes that inferable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Check the current status of a background task', a specific verb and resource that makes the tool's purpose immediately clear. It also distinguishes itself from siblings by explicitly linking to run_task with background=true and enumerating the exact statuses it returns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage context: 'use after run_task with background=true'. It does not explicitly name alternatives or conditions when not to use, but the sibling context and the status list imply when check_task applies versus reply_task or cancel_task.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.