Server Quality Checklist

Tool Scores

• analyze_pcap_fileC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions that 'LLMs control all analysis parameters' and 'can use saved configurations', but fails to describe critical behaviors like whether this is a read-only analysis, what happens with invalid inputs, performance implications for large files, or error handling. This leaves significant gaps for a tool with 6 parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences that directly address the tool's functionality. It's front-loaded with the core purpose and efficiently mentions key capabilities without unnecessary elaboration, though it could be slightly more structured for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (6 parameters, no annotations, no output schema), the description is insufficient. It doesn't explain what the analysis returns, how results are structured, error conditions, or performance considerations. For a tool that analyzes network capture files—a potentially complex operation—this leaves too many contextual gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema, mentioning 'filters, output formats, and custom fields' and 'saved configurations' which map to parameters but don't provide additional semantic context. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('analyze') and resource ('a local pcap/pcapng file'), making the purpose evident. However, it doesn't explicitly differentiate from sibling tools like 'start_capture_session' or 'manage_config', which would require a more specific comparison to achieve a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings (e.g., 'start_capture_session' for live capture vs. file analysis) or any prerequisites. It mentions 'saved configurations' but doesn't explain when they're appropriate, leaving usage context unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• stop_capture_sessionB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions that the tool stops a session and analyzes packets with LLM-controlled parameters, but lacks details on critical behaviors: whether stopping is reversible, if analysis occurs automatically upon stopping, error handling for invalid inputs, or performance implications. This is a significant gap for a tool with mutation and analysis functions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences that are front-loaded: the first states the core action, the second adds control details, and the third mentions configuration use. Each sentence adds value without redundancy, though it could be slightly more structured by explicitly separating stopping and analysis aspects.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (stopping and analyzing packets with 6 parameters), no annotations, and no output schema, the description is incomplete. It doesn't explain what happens after stopping (e.g., session deletion, data retention), the analysis output format or structure, or error conditions. For a mutation tool with rich parameters, this leaves too many unknowns for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema by mentioning 'LLMs control all analysis parameters including display filters and output formats' and 'Can use saved configurations,' which loosely relates to parameters like 'displayFilter,' 'outputFormat,' and 'configName.' This meets the baseline of 3 when schema coverage is high.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Stop a running capture session and analyze packets.' It specifies the verb ('stop'), resource ('capture session'), and additional action ('analyze packets'), which distinguishes it from siblings like 'start_capture_session' (starting vs. stopping) and 'analyze_pcap_file' (live session vs. file analysis). However, it doesn't explicitly differentiate from 'manage_config' regarding configuration usage.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by mentioning 'running capture session' and 'saved configurations,' suggesting it should be used after starting a session and potentially with configurations. It doesn't explicitly state when to use this tool vs. alternatives like 'analyze_pcap_file' for file-based analysis or 'manage_config' for configuration management, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• manage_configB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While it mentions the four operations, it doesn't describe what happens during each action (e.g., whether 'delete' is permanent, whether 'save' overwrites existing configs, error conditions, or persistence mechanisms). For a CRUD-style tool with multiple operations, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences that efficiently state the tool's purpose and value proposition. It's front-loaded with the core operations and avoids unnecessary elaboration. Every sentence contributes meaning without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 4 parameters (including a complex nested object), 100% schema coverage, and no output schema, the description provides adequate but minimal context. It covers the 'what' but lacks details about operational behavior, error handling, and integration with sibling tools. The absence of annotations increases the need for more comprehensive description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond what's in the schema - it mentions 'capture and analysis parameters' which aligns with the config object properties, but provides no additional semantic context about parameter relationships or usage patterns.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs (save, load, list, delete) and resource (reusable filter configurations). It distinguishes from sibling tools by focusing on configuration management rather than packet capture or analysis operations, making the scope immediately apparent.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions 'allows LLMs to store commonly used capture and analysis parameters for easy reuse' which provides some context, but offers no explicit guidance on when to use this tool versus alternatives like manually configuring parameters each time. There's no mention of prerequisites, limitations, or comparison with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_capture_sessionA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that this starts a 'background' session (implying asynchronous operation) and mentions 'safety limits' like packet maximums, but it doesn't cover critical behavioral aspects such as authentication requirements, rate limits, error conditions, or what happens to existing sessions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by supporting details. Both sentences earn their place by clarifying scope and capabilities without redundancy, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (starting a background capture with 6 parameters) and no annotations or output schema, the description is moderately complete. It covers the purpose and key features but lacks details on behavioral traits, error handling, and output expectations, which are important for such an operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 6 parameters. The description adds minimal value beyond the schema by mentioning 'filters, interfaces, and packet limits' and 'saved configurations,' but it doesn't provide additional semantic context or usage examples for the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Start a background packet capture session') and resource ('packet capture session'), distinguishing it from sibling tools like 'analyze_pcap_file' (analysis) and 'stop_capture_session' (termination). It also mentions key capabilities like controlling parameters and using saved configurations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (starting captures with parameter control), but it doesn't explicitly state when not to use it or name alternatives. For example, it doesn't contrast with 'manage_config' for configuration tasks or specify prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.