Server Quality Checklist

Tool Scores

• run_drgn_commandC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states what the tool does without disclosing behavioral traits. It doesn't mention whether this is read-only or destructive, what permissions are needed, how errors are handled, or any rate limits. The examples hint at functionality but lack operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with one sentence and helpful examples. It's front-loaded with the core purpose and wastes no words, though it could benefit from slightly more structure to separate examples from the main statement.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters with no schema descriptions, no annotations, and an output schema exists, the description is minimally adequate but incomplete. It covers the basic purpose and provides examples for the 'command' parameter, but doesn't address the other parameters or behavioral aspects, leaving significant gaps for a tool that executes code.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage and 3 parameters, the description adds minimal value beyond the schema. It mentions 'drgn Python code' which relates to the 'command' parameter, but doesn't explain what 'session_id' or 'truncate' do, their formats, or provide examples for them. The description doesn't compensate for the schema's lack of documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Runs drgn Python code' which is a clear verb+resource combination, but it doesn't distinguish this from sibling tools like 'run_crash_command' or explain what drgn is. The examples help but don't fully clarify the scope or differentiate from alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives like 'run_crash_command' or 'start_session'. The description implies usage for drgn code execution but offers no context about prerequisites, dependencies, or appropriate scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_crash_commandC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool runs commands but lacks critical details: it doesn't specify if this is a read-only or destructive operation, what permissions are required, how output is handled (though an output schema exists), or any rate limits. The examples hint at functionality but don't clarify behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose and includes helpful examples. There is no wasted verbiage, making it appropriately sized and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (running system commands), lack of annotations, and 0% schema description coverage, the description is incomplete. It doesn't address safety, permissions, or parameter meanings beyond command examples. However, the existence of an output schema reduces the need to explain return values, providing some baseline adequacy.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for undocumented parameters. It only mentions the 'command' parameter through examples (e.g., sys, bt), but provides no context for 'session_id' or 'truncate'. This leaves two of three parameters without semantic explanation, failing to adequately compensate for the low coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Runs') and target ('crash utility command'), and provides specific examples of commands (sys, bt, log, ps, files, vm, kmem). This makes the purpose specific and understandable, though it doesn't explicitly differentiate from sibling tools like 'run_drgn_command'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'run_drgn_command' or 'list_crash_dumps', nor does it mention prerequisites such as needing a session (implied by the session_id parameter). Usage is implied through examples but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_sessionC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states that the tool 'Starts analysis session' and 'Returns session ID', which implies a stateful operation and output format. However, it lacks details on permissions, side effects (e.g., if it consumes resources or locks files), error handling, or session lifecycle implications, which are critical for a tool that initiates analysis.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences that directly state the purpose, requirements, and return value. Every word earns its place, and it's front-loaded with the core action, making it efficient and easy to parse without any fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (starting a stateful analysis session with 4 parameters), no annotations, and an output schema (which likely covers the session ID return), the description is minimally adequate. It covers the basic purpose and key inputs but lacks depth on behavioral aspects and optional parameters, making it incomplete for safe and effective use without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It mentions 'vmcore and vmlinux paths' for the two required parameters, adding some meaning beyond the schema's bare titles. However, it omits the optional 'ssh_host' and 'ssh_user' parameters entirely, and doesn't explain what these paths represent or their expected formats, leaving significant gaps in parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Starts analysis session') and the resource ('analysis session'), which is specific and understandable. However, it doesn't differentiate from sibling tools like 'run_crash_command' or 'run_drgn_command' which might also involve analysis sessions, leaving some ambiguity about its unique role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions that it 'Requires vmcore and vmlinux paths', which provides basic prerequisites but no guidance on when to use this tool versus alternatives like 'run_crash_command' or 'stop_session'. There's no explicit when/when-not usage or comparison to siblings, leaving the agent to infer context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_sys_infoB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool runs a 'sys' command, hinting at execution behavior, but lacks details on permissions, side effects, rate limits, or output format. This is inadequate for a tool that likely interacts with system data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that clearly states the purpose without unnecessary details. It's front-loaded and wastes no words, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool likely returns system info, the presence of an output schema helps, but the description lacks context on complexity, such as what specific info is retrieved or how it differs from siblings. With no annotations and minimal description, it's adequate but has clear gaps in guiding usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter (session_id) with 0% description coverage, and the description doesn't mention parameters at all. Since there are no parameters documented in the schema, the baseline is 4, as the description doesn't need to compensate for missing param info but also doesn't add any value regarding parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool gets system info and mentions it runs a 'sys' command, which clarifies the action and resource. However, it doesn't distinguish this from sibling tools like 'list_crash_dumps' or 'run_crash_command', which might also provide system-related information, leaving the scope vague.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description mentions it's a 'convenience tool', implying it might be simpler or faster, but it doesn't specify contexts, prerequisites, or exclusions relative to siblings like 'run_drgn_command' for deeper system analysis.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_crash_dumpsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the action ('scans') and scope ('recursive'), but doesn't describe what constitutes a crash dump (file extensions, formats), whether it requires permissions, rate limits, output format (though output schema exists), or error handling (e.g., invalid paths). For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's front-loaded with the core action and resource, and the parenthetical '(recursive)' adds necessary detail without verbosity. Every word earns its place, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (file scanning), no annotations, and an output schema (which handles return values), the description is partially complete. It covers the purpose and parameter context but lacks behavioral details (e.g., permissions, error handling) and usage guidelines. With output schema existing, it needn't explain return values, but other gaps remain, making it adequate but with clear room for improvement.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal semantics beyond the input schema: it clarifies that 'search_path' is for scanning 'in the specified directory (recursive)'. However, with 0% schema description coverage (schema lacks descriptions for the parameter), the description doesn't fully compensate—it doesn't explain path format, default behavior, or validation rules. With 1 parameter, the baseline is 4, but the description provides only basic context, so a 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('scans for') and resource ('crash dumps') with specific scope ('in the specified directory (recursive)'). It distinguishes from siblings like 'get_sys_info' or 'run_crash_command' by focusing on file discovery rather than system info or command execution. However, it doesn't explicitly differentiate from all siblings (e.g., 'start_session' is clearly different).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., directory existence), when-not-to-use scenarios, or comparisons to sibling tools like 'run_crash_command' for analyzing dumps. The context is implied (scanning for files) but lacks explicit usage rules.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• stop_sessionB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. 'Terminates' implies a destructive mutation, but it doesn't disclose behavioral traits such as whether termination is reversible, what happens to session data, permission requirements, or error conditions (e.g., if session_id is invalid). The description is minimal and lacks critical context for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It is front-loaded with the core action ('Terminates') and resource ('an active session'), making it easy to scan. Every word earns its place, and there is no redundant or verbose language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (destructive mutation with 1 parameter) and no annotations, the description is incomplete—it lacks behavioral details and parameter guidance. However, an output schema exists, so return values needn't be explained. The description covers the basic purpose but falls short on usage and transparency, making it minimally adequate but with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the schema provides no parameter descriptions. The description adds no information about the 'session_id' parameter (e.g., what it is, how to obtain it, default behavior if null). However, with only 1 parameter and a default of null, the baseline is 3 as the tool might operate on a current or default session without explicit input.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Terminates an active session' clearly states the action (terminates) and resource (active session). It distinguishes from siblings like 'start_session' (opposite action) and 'get_sys_info' (different resource). However, it doesn't specify what type of session (e.g., debugging, system) or what termination entails beyond the basic verb.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., requires an active session), exclusions (e.g., don't use if session is already stopped), or comparisons to siblings like 'run_crash_command' (which might operate within a session). Usage is implied from the name but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.