Server Quality Checklist

Tool Scores

• add_to_whitelistC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action ('Add') but doesn't explain what happens upon invocation—e.g., whether it's a mutation, requires permissions, has side effects like notifications, or returns confirmation. This leaves significant gaps for a tool that modifies a security list.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste—it directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of modifying a security whitelist, no annotations, and no output schema, the description is incomplete. It doesn't cover behavioral aspects like mutation effects, error conditions, or return values, which are crucial for safe tool invocation in this context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (command, securityLevel, description) with details like enum values for securityLevel. The description adds no additional meaning beyond the schema, resulting in the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Add') and resource ('command to the whitelist'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'update_security_level' or 'remove_from_whitelist', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives like 'update_security_level' or 'approve_command'. The description lacks context about prerequisites, such as whether the command must be pending or already exist, leaving usage unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• approve_commandC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. 'Approve' implies a mutation that changes state, but the description doesn't clarify what happens after approval (e.g., does it trigger execution, log the action, or require additional steps?), potential side effects, permission requirements, or error conditions. This leaves significant gaps for a tool that likely involves security-sensitive operations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with no wasted words. It front-loads the core action and resource efficiently, making it easy to parse and understand at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, and the tool's likely role in a security/command workflow (inferred from sibling tools), the description is insufficient. It doesn't explain the outcome of approval, how it interacts with other tools (e.g., 'execute_command'), or any system constraints, leaving the agent with incomplete context for safe and effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the single parameter 'commandId' clearly documented in the schema. The description adds no additional parameter semantics beyond implying that 'commandId' refers to a pending command, which is already inferred from the tool's purpose. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('approve') and the target resource ('a pending command'), which is specific and unambiguous. However, it doesn't differentiate this tool from its sibling 'deny_command' beyond the opposite action, nor does it explain what 'approve' entails in this context versus simply executing or modifying the command.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'deny_command' or 'execute_command'. It doesn't specify prerequisites (e.g., that the command must be in a pending state) or contextual cues for selection, leaving the agent to infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deny_commandC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the action ('deny') but doesn't explain what 'deny' entails—whether it's reversible, if it requires specific permissions, what the outcome is, or if there are side effects. For a mutation tool with zero annotation coverage, this leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core action and target, making it easy to parse quickly. Every word earns its place by conveying essential information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that this is a mutation tool with no annotations and no output schema, the description is incomplete. It doesn't explain the tool's behavior, outcomes, or error conditions. The agent lacks context on what 'deny' means operationally, making it difficult to use correctly without additional information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with clear documentation for both parameters ('commandId' and 'reason'). The description doesn't add any meaning beyond what the schema provides, such as format examples or constraints. Since schema coverage is high, the baseline score of 3 is appropriate, as the schema handles parameter semantics adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('deny') and the target ('a pending command'), providing a specific verb+resource combination. However, it doesn't differentiate from sibling tools like 'approve_command' or explain what 'deny' means in this context beyond the basic action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'approve_command' or 'execute_command'. It doesn't mention prerequisites (e.g., needing a pending command ID) or context about what happens after denial. The agent must infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_commandC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states what the tool does but doesn't describe critical behavioral aspects: whether execution is immediate or requires approval, what permissions are needed, whether commands run with elevated privileges, what happens on failure, or what output format to expect. For a potentially dangerous shell execution tool, this is inadequate transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that communicates the core functionality without unnecessary words. It's appropriately sized for a tool with two parameters and gets straight to the point. Every word earns its place, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of shell command execution (potentially destructive, security-sensitive) and the absence of both annotations and output schema, the description is insufficiently complete. It doesn't address safety considerations, error handling, output format, or how this interacts with the security-focused sibling tools. For a tool that could modify system state, more contextual information is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters ('command' and 'args') clearly documented in the schema. The description doesn't add any parameter-specific information beyond what the schema already provides. According to scoring rules, when schema coverage is high (>80%), the baseline is 3 even without additional param details in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Execute') and target ('a shell command on macOS'), making the purpose immediately understandable. It distinguishes from siblings by focusing on command execution rather than security management or querying. However, it doesn't specify whether this executes immediately or requires approval, which could be more precise.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives like 'approve_command' or 'deny_command'. The description doesn't mention prerequisites, security implications, or whether this bypasses approval workflows. With siblings focused on security controls, this omission leaves significant ambiguity about appropriate usage contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• remove_from_whitelistC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the tool performs a removal action, implying mutation, but doesn't describe permissions needed, whether the change is reversible, error conditions, or what happens if the command isn't in the whitelist. This leaves significant gaps for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that states the core function without any wasted words. It's appropriately sized for a simple tool with one parameter and gets straight to the point.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description is insufficient. It doesn't explain what happens after removal (e.g., confirmation message, error responses), doesn't mention security implications, and provides no context about the whitelist system it operates within.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single 'command' parameter adequately. The description doesn't add any additional meaning about parameter format, constraints, or examples beyond what the schema provides, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('remove') and target resource ('a command from the whitelist'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'deny_command' or 'update_security_level' which might have overlapping security functions, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'deny_command' or 'update_security_level'. The description only states what it does, not when it's appropriate or what prerequisites might exist (e.g., whether the command must already be whitelisted).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_security_levelC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While 'Update' implies a mutation, it doesn't specify required permissions, whether changes are reversible, error conditions (e.g., if command isn't whitelisted), or side effects. This leaves significant gaps for a security-related mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with zero wasted words. It front-loads the core action and target efficiently, making it easy to parse while conveying essential purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description is insufficient. It lacks critical context: what 'security level' means operationally, how changes affect command execution, error handling, or response format. Given the security-sensitive nature and sibling tools, more completeness is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are fully documented in the schema. The description adds no additional parameter context beyond implying 'command' refers to a whitelisted one. This meets the baseline for high schema coverage but doesn't enhance understanding of parameter usage or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Update') and the target ('security level of a whitelisted command'), making the purpose immediately understandable. However, it doesn't explicitly differentiate this tool from siblings like 'add_to_whitelist' or 'remove_from_whitelist', which also modify command permissions but in different ways.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., the command must already be whitelisted), contrast with 'approve_command'/'deny_command' for pending requests, or explain when changing security levels is appropriate versus adding/removing from the whitelist.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_pending_commandsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states this is a read operation ('Get'), implying it's non-destructive, but doesn't cover other aspects like permissions needed, rate limits, response format, or whether it returns all pending commands or a filtered subset. The description is minimal and lacks behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It's front-loaded with the core purpose and efficiently communicates the tool's function without unnecessary elaboration, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no annotations, no output schema, and a simple input schema with no parameters, the description is minimal. While it states the purpose, it lacks context about what 'pending approval' means, how commands are structured, or what the return value looks like. For a tool in a security/command approval context, more completeness would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so no parameter documentation is needed. The description doesn't add parameter details, which is appropriate here. A baseline score of 4 is given since the schema fully covers the absence of parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('list of commands pending approval'), making the purpose immediately understandable. It doesn't explicitly distinguish from sibling tools like 'approve_command' or 'deny_command', but the verb 'Get' implies a read operation rather than an action on the commands themselves.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'get_whitelist' or 'execute_command'. It doesn't mention prerequisites, context for pending commands, or any exclusions, leaving the agent to infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_whitelistB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states 'Get the list' which implies a read-only operation, but doesn't specify whether this requires authentication, returns paginated results, includes metadata, or has rate limits. For a tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It's front-loaded with the core purpose ('Get the list of whitelisted commands') and doesn't include unnecessary elaboration. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and no output schema, the description is minimally adequate but lacks context about the return format (e.g., structure of the whitelist, data types) and behavioral traits. It meets the basic requirement for a simple read operation but doesn't provide enough information for confident use without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema fully documents the absence of inputs. The description doesn't need to add parameter details, and it appropriately doesn't mention any. Baseline 4 is correct for zero-parameter tools where the schema already covers everything.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and the resource ('list of whitelisted commands'), making the purpose immediately understandable. It doesn't explicitly differentiate from siblings like 'get_pending_commands' or 'add_to_whitelist', but the verb+resource combination is specific enough to infer basic distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, timing, or contrast with sibling tools like 'get_pending_commands' (which might retrieve unapproved commands) or 'add_to_whitelist' (which modifies the list). Usage is implied by the name but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.