Server Quality Checklist

Tool Scores

• send_event_to_external_input_jobA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must fully convey behavioral traits. It mentions that Arcanna generates a new internal ID, but it does not disclose potential side effects, required permissions, rate limits, or the impact of sending data. The error-handling note is helpful but insufficient for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the purpose, followed by an error-handling note, then a parameter list. It is relatively concise, though the 'Parameters:' header and empty line add slight redundancy. The structure is clear but could be tighter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description includes return value details despite no output schema, which is good. However, it does not specify the expected format or keys for the event dictionary, leaving ambiguity for nested objects. It also fails to differentiate from sibling tools or explain prerequisites. Overall, adequate for a simple tool but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description adds meaning: job_id as 'Unique identifier for the external input job' and event as 'A raw dictionary containing event/alert/incident data.' This compensates for the missing schema descriptions and provides clarity beyond the titles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: send a JSON event payload to Arcanna for a decision, generating a new internal ID. This distinguishes it from the sibling tool 'send_event_with_id_to_external_input_job', which presumably takes an existing ID. The verb and resource are specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  While the description provides an error-handling instruction ('do not use any other tool, ask the user'), it lacks explicit guidance on when to use this tool versus the sibling tools. It implies use for new events without an ID, but this is not directly stated. No exclusions or alternatives are described.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• send_event_with_id_to_external_input_jobA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states that the tool sends data to Arcanna and returns a dict with status and error_message, implying mutation and potential failure. However, it does not disclose authorization requirements, rate limits, idempotency, or whether it is destructive. The description is adequate but not thorough given the lack of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear summary sentence followed by 'Parameters:' and 'Returns:' sections using bullet points. It is front-loaded with the main purpose. However, the parameter descriptions are redundant with the argument list (though needed due to schema deficiencies), and the return section could be slightly more concise. Still, it earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains all parameters and the return value, which is helpful. However, it assumes the agent knows that job_id must correspond to an existing external input job (presumably retrievable via 'get_external_input_jobs'), and it does not mention any dependencies or setup steps. The return value includes error_message but no explanation of error handling. Given the tool's complexity (3 required params, nested object), it is partially complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, meaning it provides no descriptions for parameters. The tool's description explicitly describes each parameter (job_id as int referencing an external input job, event as raw dict, event_id as unique identifier), adding significant meaning beyond the schema. It also details the return value structure, which is absent from any output schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Send') and the resource ('a JSON security alert/incident/event to Arcanna for ingestion'). It distinguishes from the sibling 'send_event_to_external_input_job' by including 'with_id' in the name and describing the event_id parameter, indicating this variant allows sending with a specific event identifier.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus the sibling 'send_event_to_external_input_job' (without event_id). It lacks context on prerequisites (e.g., needing to obtain a job_id from 'get_external_input_jobs') or when not to use it. No alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• health_check_input_serverA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility. It explains the return dict and the meaning of the 'status' field, which helps the agent understand outcomes. However, it does not disclose potential side effects, auth requirements, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise: two short paragraphs with no wasted words. The purpose is front-loaded, and the return details are clearly separated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple health check tool with no parameters and no output schema, the description sufficiently explains the return value. It could mention authentication implications, but the current level is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description does not need to add parameter-level information. The baseline score of 4 is appropriate as it is not missing any parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Health check' and the resource 'Arcanna Input API Server'. This distinguishes it from sibling tools like get_external_input_jobs or send_event_to_external_input_job, which deal with jobs and events.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose is self-evident as a health check, but there is no explicit guidance on when to use it versus alternatives, nor any when-not conditions. The description implies usage for verifying server status but lacks clarity on decision boundaries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_external_input_jobsA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It describes the return type and fields in detail, implying a read operation. However, it does not explicitly state the absence of side effects, auth requirements, or potential errors (e.g., empty result set).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with bullet points and front-loaded purpose. Slightly verbose in the return list but each element adds value. Could be slightly more concise, but effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Comprehensive for a zero-parameter tool with no output schema. Provides detailed return field descriptions, clear usage guidelines, and sufficient context to understand when to invoke.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters, schema_description_coverage is 100%, so baseline is 4. The description adds context about what the tool returns, which is appropriate since the schema is empty.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Retrieve Arcanna External Input Jobs' and explains the specific resource. It distinguishes from sibling tools by specifying when to use, using specific verb and resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use scenarios with examples, and clearly lists cases where the tool should NOT be used, directing to a different tool for general job listing. This helps the agent avoid misuse.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.