Server Quality Checklist

Tool Scores

• get_capability_graphA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It clearly describes the return payload (nodes, edges with danger flag, attack paths, risk score), which provides useful transparency. However, it does not disclose potential side effects, prerequisites such as agent existence, or performance/caching behavior, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the intent and packs all key return elements into a compact list. Every word contributes useful information, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple one-parameter schema and absence of an output schema, the description provides a reasonably complete picture of what the tool returns. It covers the main output components and overall risk score, though it could mention error behavior or valid agent ID requirements for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage for the single parameter agentId with the description 'ID of the agent to inspect'. The tool description adds no additional parameter-level detail, so the baseline of 3 applies since the schema fully documents the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the primary action ('Return') and the specific resource ('the agent's capability graph'), enumerating its key components (nodes, edges, attack paths, risk score). This differentiates it from siblings like detect_attack_paths and apply_policy_fix, which suggest different operations (detection and modification).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for retrieving graph-based attack-path and risk information for a given agent, but it does not explicitly state when to choose this over sibling tools like detect_attack_paths. No exclusions or alternative tool references are provided, so usage guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• detect_attack_pathsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool is deterministic and returns severity and risk score, which adds behavioral context. However, it does not explicitly state side effects (e.g., read-only nature), permissions, or potential impacts, leaving some ambiguity for a detection tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the action and output. Every word adds value, and there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (one parameter, no output schema), the description adequately covers what it returns. It does not explain prerequisites or integration with sibling tools, but for a straightforward detection operation with clear output, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with agentId described as 'ID of the agent to inspect.' The description's 'for an agent' reiterates this but adds no additional parameter-level detail, so it meets the baseline without exceeding it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: running a deterministic toxic-capability-combination detector for a given agent and returning attack paths with severity and an overall risk score. This specific verb-resource pairing and output details distinguish it from sibling tools like get_capability_graph or apply_policy_fix.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool: when you need to detect attack paths for an agent. It does not explicitly mention alternatives or exclusions, but the context is clear enough to infer its purpose relative to siblings. No contradicting usage guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• connect_toolA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It states the effect (grants capabilities) and the return value (updated capability set), which is helpful. However, it omits details about side effects, reversibility, prerequisites, or what happens if the tool is already connected, leaving some transparency gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the verb and resource, and zero waste. The description is concise while covering purpose, effect, and return value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two well-documented parameters and no output schema, the description adequately explains the operation and explicitly notes the return value. It provides enough context for correct use without needing further detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema descriptions cover 100% of the parameters, with toolId already listing the exact allowed values and agentId being clearly explained. The description adds no extra parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: connecting a third-party tool to an agent and granting its capabilities. It lists specific example tools and explicitly distinguishes this tool from the sibling tools (capability graph, attack paths, policy fix) by focusing on the connection action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context on what the tool does and its effect (grants capabilities), making it obvious when to use it over siblings. However, it does not explicitly mention when not to use it or what alternatives exist, but the context is sufficient for basic selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• apply_policy_fixA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the mutating effect (disconnecting a tool) and the return value (refreshed capability graph). This is transparent, though it doesn't discuss reversibility or failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is direct and well-structured, using 'then' to indicate sequence. No redundant words, front-loaded with the main action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and lack of output schema, the description covers the essential behavior and return. It could mention prerequisites or error states, but these are not necessary for basic invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear parameter descriptions. The tool description adds no extra parameter-specific meaning beyond what the schema already states, so baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the action: 'Remediate a detected attack path' with a precise mechanism ('disconnecting the tool that supplies its riskier capability'). It also mentions returning the refreshed graph, distinguishing it from siblings like connect_tool and get_capability_graph.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after an attack path is detected, providing clear context. It does not explicitly exclude alternatives, but the purpose is distinct enough to guide selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.