Server Quality Checklist

Tool Scores

• explain_selectionA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden here. It is transparent about the behavior: included symbols are returned with scores and token costs, and dropped symbols are scored but cut by the token budget. It does not explicitly state side effects or authorization requirements, but its read-only nature is strongly implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, then the return behavior, then a compact Args block. Every sentence adds value; there is no filler or unrelated detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no annotations, the description covers the essential aspects needed to invoke the tool: what it does, what it returns, and what each parameter means. The only notable gap is the lack of comparative routing guidance against the sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema property descriptions have 0% coverage, so the description must compensate. It does: repo_path is 'Absolute path to the repository' with a fallback to the --repo startup value, symbol is described as 'The changed symbol ID to build context for,' and max_tokens is defined as a 'Token budget' with a default. This adds meaningful semantics beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb and resource: 'Explain why symbols were included or dropped from context.' It also details what is returned. It does not explicitly compare itself to siblings such as compile_context or verify_retrieval, but the purpose is specific enough to be distinguished from them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: after context selection, an agent can 'inspect or filter the selection.' It also explains parameter defaults such as using the startup repo when repo_path is omitted. However, it does not explicitly state when to use this tool over its siblings or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compile_contextA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It exposes useful behavioral details: output is 'packed into max_tokens', a disclosure headers shows what was dropped, retrieval can be biased by task_description, and the pass@1 effect of the meta level is explicitly marked 'UNMEASURED'. These are substantive disclosures, not merely rephrased schema information.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core action and then expands into concrete input formats, defaults, and caveats. It contains no filler, yet it is rather long due to the detailed Args block. Every sentence earns its place, and the most critical behavioral caveat ('UNMEASURED') is included without unnecessary qualification.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 6-parameter tool with no annotations and no input-schema descriptions, the description covers all required behavior: input formats, defaults, output composition, token budget, disclosure header, and task bias. The one gap is explicit behavior when all inputs are omitted: it only explains defaulting to HEAD when 'task_description' is given, not what happens when no parameters at all are passed. This is a small but real completeness gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although input-schema content coverage is 0%, the description's Args block thoroughly documents all six parameters with examples, defaults, mutual exclusivity, and behavioral caveats. It explains the exact meaning of changed_symbols and git_ref, the default git_ref when only task_description is passed, and the enum-like behavior of meta. This fully compensates for the sparse schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence states a specific verb and resource: 'Compile LLM-ready context for a change.' It further clarifies the resource by naming the accepted inputs (changed symbol IDs or git ref) and the returned content (callers, callees, related functions). It does not explicitly differentiate from the sibling tools, but the purpose is clear enough to be usable in isolation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong input-selection guidance: changed_symbols vs git_ref are mutually exclusive, and task_description alone implies HEAD. However, it says nothing about when to use compile_context over its siblings (find_impact, explain_selection, verify_retrieval), so the when-to-use guidance is limited to parameter choices rather than tool-disambiguation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_impactA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the behavioral burden. It clearly discloses the output: direct callers, direct callees, transitive impact, and even describes behavior when repo_path is omitted, using the server startup --repo. It does not explicitly call itself read-only, but the language 'Find... Returns' strongly implies a query-only operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short, front-loaded with the core purpose, and each sentence earns its place. The summary 'The "what breaks if I change this" query' is a useful clarifying mental model rather than unnecessary repetition. The two argument descriptions are compact and relevant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a fairly simple 2-parameter lookup tool, the description covers the inputs, the core behavior, and the scope of results, and an output schema exists to fill in any return-format uncertainty. It doesn't go quite further to explicitly mention read-only constraints, staleness, or required indices, but there are no annotations and the description is still sufficiently rich for most AI agents.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 0% description coverage, but the description fully compensates for both parameters: repo_path is defined as an absolute path with the startup --repo fallback, and symbol is given an exact Symbol ID example (./src/auth.py:validate_jwt). This adds significant, actionable detail beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: finding what breaks if a symbol is changed, and what it returns: direct callers, direct callees, and transitive impact. It is specific enough to be distinguished from generic helpers, but it does not explicitly differentiate from sibling tools like compile_context or verify_retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase "The 'what breaks if I change this' query" gives a strong, useful usage frame: call this when you are about to change a symbol and want to assess impact. However, there is no explicit statement about when not to use it, no comparison to alternatives, and no exclusions, so the guidance remains implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_retrievalA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Even though no annotations were provided, the description reveals significant behavioral traits: it generates test cases from co-change history, runs DiffContext retrieval, reports hit/recall, and optionally prints NULL RESULT when the repo doesn't fit. Calling out the NULL RESULT as a feature is genuinely useful and goes beyond typical descriptions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The definition is short, front-loaded with the main purpose, then narrows to mechanism and parameters. The NULL RESULT sentence is pointed and earns its place. The Args section provides compact parameter context without re-describing what the schema already defaults.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is an output schema and only two simple parameters, the description is complete: it covers what the tool does, the algorithm, output metrics, edge-case behavior, and all parameter semantics. An agent has enough information to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries all param documentation. It clearly adds meaning by explaining repo_path as an absolute path with a startup fallback to --repo, and describes n as the maximum number of test cases with a default of 20. This covers both parameters meaningfully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Mine git history and grade retrieval quality on your repo,' and then defines the mechanism (generate test cases, run retrieval, report hit/recall). It clearly explains what the tool does, but it does not explicitly contrast it with sibling tools like compile_context or find_impact.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives clear context: you use this when you want to assess retrieval quality over git history, and the NULL RESULT warning indicates what to expect if the repository isn't a good fit. However, it does not state explicit when-to-use versus known sibling alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.