Server Quality Checklist

Tool Scores

• reaching_definitionsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It states the analysis is done statically and describes the output as a def-use graph. However, it does not disclose error handling (e.g., missing function) or performance implications. The return structure is outlined but side effects are not mentioned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose and includes an example usage. It is structured with paragraphs, args, and returns. A bit verbose but each sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description provides an expected return format. However, it lacks details on error cases, limitations (e.g., large functions), or prerequisites. For a non-trivial analysis tool, more completeness would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description adds essential meaning: 'path: PE / ELF / MachO' and 'function: function name (e.g., "main")'. While helpful, it does not specify constraints like file format validation or path requirements.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Compute the reaching-definitions graph for *function*' and explains what reaching definitions are with a concrete example for the re-mba-deobfuscate skill. It distinguishes itself from sibling tools like build_cfg and symbolic_exec by focusing on dataflow analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions it is 'Useful for the re-mba-deobfuscate skill' but does not explicitly specify when to use or not use this tool versus alternatives. No exclusions or comparisons to siblings are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• build_cfgA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided so description carries full burden. It describes the return format and use case but omits performance, limitations, or error conditions. Partially adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the main action, and structured with Args/Returns sections. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations or output schema, the description covers purpose, parameters, and return but lacks details on side effects, permissions, or prerequisites. Adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates by explaining 'path' as PE/ELF/MachO and 'function' as optional with default None behavior, plus return format. Adds significant meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it builds a control-flow graph of a binary path, optionally for a single function. This distinguishes it from sibling tools like symbolic_exec and reaching_definitions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains when to use the CFG (first thing for VM detection, binary comprehension) but does not explicitly state when not to use or mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_angrA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description thoroughly explains the fallback chain, the import probe logic, the distinction between CLI and Python import, and the conditions for OK vs WARN status, which is highly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is detailed but well-structured, with clear explanations. It could be slightly more concise, but every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately explains the return status (OK/WARN) and provides concrete install hints, making it complete for its purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, and the schema coverage is 100%. The description does not need to add parameter info, meeting the baseline for no parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns version info and import probe results. It distinguishes from siblings like build_cfg, reaching_definitions, and symbolic_exec by being a health check for angr dependencies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool should be used before running other angr tools to ensure dependencies are available, but it does not explicitly state when to use it or mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• symbolic_execA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description fully carries the burden of behavioral disclosure. It explains the tool is a partial trace, explores until timeout or all paths, describes return values, and notes default behavior for args. No annotations exist, so this is comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured as a docstring with clear sections. Every sentence adds value, including purpose, parameter details, return format, and use case. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, parameters, return format, and limitations (partial trace, timeout). It could mention default timeout or how to set it, but is generally complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds significant meaning: path is PE/ELF/MachO, address is hex string, args are optional symbolic-arg names with default behavior explained. It compensates well for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Run angr symbolic execution starting at *address*', specifying the tool and key input. It distinguishes the tool from siblings (build_cfg, check_angr, reaching_definitions) by focusing on symbolic execution and mentioning cross-validation with Triton.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for symbolic execution and cross-validation with Triton, but does not explicitly state when to use this tool over alternatives. No direct comparison or when-not-to-use guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.