Server Quality Checklist

Tool Scores

• get_compressedB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description must disclose behaviors. It does not state whether the operation is read-only or modifies files, nor does it mention side effects, auth requirements, or rate limits. The word 'compress' might imply mutation, but the return of a 'compressed structure' suggests the original file is unchanged. This ambiguity is a gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single well-structured sentence. It front-loads the action ('S5-compress a single source file'), specifies file types, details the compression behavior, and lists the output. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Without output schema, the description should explain the return value structure, but it does mention 'compressed structure plus token stats (tokens_before, tokens_after, reduction_pct),' which is helpful. However, it lacks behavioral context (read/write), does not resolve the file type inconsistency, and does not address how this relates to sibling tools like 'expand.' The minimal viable information is present, but gaps reduce completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (one parameter with description). The description adds that the tool supports Python and JS/TS, but the schema's parameter description says 'Path to a .py file,' which contradicts the broader file types stated in the tool description. This inconsistency misleads the agent and detracts from the value added beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool compresses a single source file (Python or JS/TS) by stripping bodies while keeping signatures and dependency contracts, and returns token stats. The verb 'compress' and resource 'source file' are specific, and it distinguishes from siblings like 'blast_radius' or 'expand' implicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description does not mention when to choose compression over expansion or other sibling tools, leaving the agent without decision context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• index_repoA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses maximum depth 3 and skipped directories (__pycache__, .git, venv, node_modules), but fails to mention behavior for unsupported file types, invalid paths, or performance implications for large repos. Additional context would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with front-loaded purpose and clear constraints. Every word adds value, no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description partially explains return value as 'aggregated structural index' listing functions, classes, imports, dependencies. However, it lacks details on output format (e.g., JSON structure, list vs. dict) and does not address edge cases like empty repos or mixed languages. It is acceptable but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a single required 'path' parameter described adequately. The description adds no extra parameter details beyond the schema, so baseline 3 is appropriate. The description's main value is in explaining the tool's overall function, not parameter specifics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool walks a repo and returns a structural index of functions, classes, imports, and dependencies, explicitly specifying supported languages (Python and JavaScript/TypeScript). This distinguishes it from sibling tools like 'blast_radius' or 'security_surface', which likely have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives like 'blast_radius' or 'expand'. It only implies usage for structural indexing, but lacks when-to-use, when-not-to-use, or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• expandA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so description carries full burden. It states it returns 'full source' and 'compressed dependency context', which is a key behavioral trait. However, it does not disclose read-only nature, auth requirements, or potential performance implications (e.g., large files). Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with purpose. No wasted words; all information is necessary and directly conveys the tool's action and output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is no output schema and no annotations, the description is somewhat vague about the exact format of 'compressed dependency context'. While the tool is simple, the description could be more complete by clarifying what 'compressed' means. Sufficient but leaves room for ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters adequately. The tool description adds no new meaning beyond restating the schema descriptions, meeting the baseline but not adding value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses specific verb 'expand' on function/class names in a .py file, clearly indicating it returns full source plus dependency context. Differentiates from siblings like 'get_compressed' (which likely returns compressed content) and 'blast_radius' (impact analysis).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives like 'get_compressed' or 'blast_radius'. Usage is implied by the description (to get full source of specific definitions with dependencies), but lacks when-not-to-use or prerequisite information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• security_surfaceA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It clearly states pure AST analysis, zero LLM calls, risk classification levels, secret detection, max depth 3, and skipped directories. Could mention whether the tool is read-only (likely yes) but overall quite transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is front-loaded with the main action and packed with relevant details in three sentences. Minimal redundancy, though length could be trimmed slightly without losing meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, description adequately explains return values (high_risk, medium_risk, secrets, clean files, summary). Covers key behaviors and limitations. Could mention file type filtering but sufficient for an agent to understand scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter (repo_path) with 100% schema coverage. Description adds no extra detail beyond the schema's description of the path. Baseline score is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it walks a Python repo using AST analysis to find dangerous function calls and secrets, with specific risk classifications. No sibling tool overlaps, making its unique purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (security scanning of Python repos) but does not explicitly state when not to use or mention alternatives among siblings. More precise guidance would improve this dimension.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• blast_radiusA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: BFS algorithm, inverted call edges, returns specific fields (direct_callers, all_impacted, etc.), and scope (Python files only). No hidden side effects or destructive actions are implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loads the purpose, and includes all essential information without verbosity. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations or output schema, the description covers purpose, algorithm, input parameters, return fields, and file type restriction. It is complete for an agent to understand and use this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (both params described), so baseline is 3. The description adds value by explaining the algorithm context (BFS over inverted call edges) and linking the parameters to the analysis process, though the schema already defines them clearly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Reverse-call-graph impact analysis for a Python function', specifying the verb (analyze impact), resource (Python function), and methodology (BFS over inverted call edges). It distinguishes from sibling tools like index_repo and security_surface which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly defines usage: when you need to know which functions are affected by changing a target function in a Python file. It does not explicitly compare with siblings, but the context is clear. The restriction 'Python files only' provides a clear constraint, earning a 4.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.