Server Quality Checklist

Tool Scores

• list_dex_classesA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes output structure (fqn, access, method_count) and notes it's a read-only listing of the typed class graph. No annotations provided, so description carries full burden; it adequately reveals behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with a code block showing output format and clear distinction from Apktool. Slightly verbose but each sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Explains return format and use case, but lacks description of the 'path' parameter. No output schema and no annotations, so the description is incomplete for a one-param tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has one required parameter 'path' with no description in the schema or the description. Schema coverage is 0%, and the description does not explain its purpose (e.g., path to APK or directory).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it enumerates every class in every classes*.dex in the APK, returning one record per class with specific fields. Distinct from sibling tools like decode_manifest or parse_apk.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use this for the 'what's actually compiled into this APK' view and contrasts with Apktool's Smali dump. Could be improved by naming sibling tool alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• decode_manifestA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full behavioral transparency. It explains the output format (text + parsed lists), and discloses error handling for binary-only manifests, stating that errors are surfaced rather than silently returning partial results. However, it does not discuss permissions, side effects, or read-only nature explicitly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three short paragraphs covering purpose, return value, and error handling. It avoids redundancy and is front-loaded with the main action. Minor improvement could be made by structuring the list of returned elements more compactly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter, no output schema, and no annotations, the description provides adequate context: what it does, what it returns, and how it handles failures. It lacks details on path format or prerequisites, but overall it is sufficiently complete for an experienced analyst.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one required parameter 'path' with no description in the schema (0% coverage). The tool description does not explain what 'path' refers to (e.g., file path on the system), leaving the agent to infer its meaning. This lack of detail hampers correct parameter specification.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool decodes the APK's AndroidManifest.xml and returns text and parsed component lists. It uses a specific verb ('Decode') and resource ('APK's AndroidManifest.xml'), distinguishing it from sibling tools that focus on other APK analysis tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when decoding the manifest, but does not explicitly state when to use this tool versus alternatives (e.g., parse_apk). It mentions a fallback to apktool d for binary-only manifests, providing some context, but lacks clear when-not guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• classify_apk_protectionA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It details the matching process against classes.dex, lib/*.so, and manifest, and describes return format. It is transparent about the tool's behavior, though it could explicitly state it is read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a summary, parameter details, return format, and matching targets. It is front-loaded with the purpose. Slightly verbose but every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description provides a detailed return structure including fields like category, evidence, and truncation info. It covers the tool's complexity adequately, though missing edge case handling notes.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description provides clear explanations for both parameters: 'path: path to the APK' and 'max_per_category: per-category match cap (default 50)'. This compensates well for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool walks APK and classes.dex for known protection patterns. It distinguishes from sibling tools like decode_manifest or list_dex_classes by specifying its unique function of classifying protection patterns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives. Although sibling tool names are given, the description itself lacks context for selection, which is necessary for an AI agent to choose correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_apktoolA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, description carries full burden. It explains backend selection and preference, but does not disclose whether it has side effects or makes external calls. It is adequately transparent for a read-only check.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is 5 sentences, front-loaded with main purpose. Could be slightly more concise but is well-structured and informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and no output schema, the description covers what the tool does and what it returns (availability, active backend). Sufficient for a simple check tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Tool has no parameters, so description need not add parameter info. Baseline 4 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool returns availability of apktool, androguard, and java. It explains the active backend logic and preference, distinguishing it from sibling tools that analyze APKs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies usage for checking backend availability before analysis, but lacks explicit when-to-use or when-not-to-use guidance. No alternatives mentioned, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• parse_apkA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses the use of androguard for manifest fields and a fallback to zipfile-only walk when androguard is missing. This is good transparency about dependencies and behavior, especially in the absence of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is 8 lines with a concise first sentence, followed by bulleted output details and a note on fallback. Every sentence adds necessary information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for a structural summary tool: covers input, detailed output fields, and fallback behavior. No output schema is needed given the bulleted list. One parameter is well-handled.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter (path) with 0% schema coverage, but the description adds value by specifying 'APK at *path*', clarifying it is a file path to an APK. This is sufficient given the single parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Return a structural summary of the APK at *path*' and enumerates specific fields included. It distinctly covers the full APK summary scope, differentiating it from sibling tools that focus on specific aspects (e.g., decode_manifest, classify_apk_protection).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs. siblings. While the description lists output and fallback, it does not advise on which scenarios favor parse_apk over more specific tools like decode_manifest or list_dex_classes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.