Server Quality Checklist

Tool Scores

• read_payload_chunkA
  4.2/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description states 'payload file is automatically deleted after the last chunk is read,' which contradicts annotations (readOnlyHint=true, idempotentHint=true) that imply no side effects. This is a serious inconsistency, misleading about destructiveness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (~150 words) with clear sections (use, iteration, args, returns). Every sentence earns its place, no redundancy, and the most important info (usage pattern) is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and the complex iterative workflow, the description fully explains how to use the tool in conjunction with scan_codebase, the chunking mechanism, and the file deletion. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although the schema describes parameters, the description adds critical context: project_path must match scan_codebase call and chunk_index is zero-based. It also explains the return structure, adding value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Read a chunk of the analysis payload produced by scan_codebase,' identifying the specific verb and resource. It distinguishes from siblings (generate_agents_md, scan_codebase) by focusing on payload retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'Call this tool repeatedly starting at chunk_index=0, incrementing by 1 each time, until the response contains has_more=false. Concatenate all data fields.' This leaves no ambiguity about the iterative workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_codebaseA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already include idempotentHint=true and destructiveHint=false; the description adds valuable context about writing the analysis payload to disk, caching behavior, and explicitly confirms the payload is pure architectural data with no AGENTS.md instructions. This goes beyond the annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact yet comprehensive, with a clear structure: purpose, behavior, supported languages, arguments, and return value. It is slightly verbose but every section adds value, and the front-loaded purpose sentence aids quick scanning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers all essential aspects: purpose, side effects, supported languages, parameter semantics (though flawed), and return value instructions. It is sufficiently complete for a complex tool with nested parameters and an output schema, though the parameter default inconsistency weakens completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains both parameters, but critically contradicts the schema on the default for force_full_scan: description says 'default: True' while schema states 'default: false'. This inconsistent guidance undermines the added semantics. Schema coverage is low (0% at top-level), but the nested descriptions exist, making the misstatement more harmful.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb and resource: 'Scan and analyze a codebase with tree-sitter, producing a structured context payload.' It distinguishes itself from siblings by explicitly directing users to generate_agents_md for AGENTS.md tasks and mentioning read_payload_chunk for retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use ('when you need deep codebase understanding for any task') and when-not-to-use ('To generate or update AGENTS.md specifically, use generate_agents_md instead'). Also gives conditional guidance for force_full_scan, noting when to set it to False in incremental workflows.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_agents_mdA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=false, destructiveHint=false, idempotentHint=true. Description adds context about checking file existence, return values, and workflow steps, but doesn't detail side effects beyond mutation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is well-structured with bullet points and front-loaded purpose. Could be slightly shorter but no wasted content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, description covers return shape and workflow steps adequately. Missing error handling details but sufficient for expected use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has thorough descriptions for params and project_path. Description merely repeats schema info (project_path default and purpose), adding no new meaning beyond what schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Orchestrate the full AGENTS.md creation or update workflow' and distinguishes from siblings by specifying when to use scan_codebase + read_payload_chunk instead.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this tool whenever the user asks to generate, create, update, or refresh AGENTS.md' and provides alternative tools for general context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.