Server Quality Checklist

Tool Scores

• get_component_impactA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explains what the tool does (finds modules and PRs) and adds the 'blast radius' concept, but does not disclose potential limitations, such as whether results are direct or transitive, performance implications, or authentication requirements. While useful, it lacks depth beyond the core purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded: a clear one-sentence purpose, a short usage note, and a single-parameter Args section. Every sentence earns its place with no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter, no annotations, and an output schema present, the description covers the essential aspects: what it does, when to use it, and what the component_name should look like. Minor gaps include not specifying the output's shape or edge cases, but the output schema exists and the tool is simple enough that this description is largely adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only a raw string parameter with no description, so the description must compensate. It does so effectively by explaining 'File path or service name' and providing concrete examples ('api_service.py', 'backend/app/auth'), which adds meaning beyond the schema. It could go further by clarifying matching semantics (exact vs. partial), but it still meets the need.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Find all modules and PRs where a specific component (file/service) appears'—a specific verb, resource, and scope. It further distinguishes itself from siblings by emphasizing 'blast radius' and 'what depends on it and what it depends on', making its unique purpose obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides a clear use context: 'Use this to understand the blast radius of changes to a component.' This implies when to use the tool, though it does not explicitly mention alternatives or when not to use it. The guidance is sufficient for a simple tool with a single purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_recent_changesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. 'Get' implies a read-only operation, and the description mentions sorting by recency ('most recently'), but it does not explicitly state that it has no side effects, nor does it disclose any auth needs, rate limits, or return format details. The output schema may cover return structure, but the description adds minimal behavioral context beyond the verb.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is five lines with a clear front-loaded purpose, followed by a compact 'Useful for' note and a structured Args list. Every sentence contributes: the first states the action, the second provides usage context, and the Args define parameters. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (two optional params, no nested objects, and an output schema present), the description covers the essential context: purpose, parameter semantics, and a use case. It lacks explicit guidance on how the 'entire codebase' scope is determined or what exactly qualifies as a 'fact,' but the output schema likely addresses the latter. The description is nearly complete for this type of list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no descriptions (0% coverage), so the description must compensate. The Args section fully defines both parameters: 'category' as an optional filter with enumerated values ('behavior', 'design', or 'assumption'), and 'limit' as the number of facts with a default of 10. This goes well beyond the raw schema by providing meaning and allowed values.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource+scope: 'Get the most recently extracted facts across the entire codebase.' This clearly distinguishes it from siblings like get_module_facts (module-specific) and search_codebase (search). The 'recently' qualifier uniquely identifies this tool's role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when it's useful: 'for understanding what changed recently or getting an overview of the codebase.' This provides clear usage context. However, it does not mention alternative tools or explicitly say when not to use it, leaving no exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modulesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It discloses the divergent behavior based on the parameter (compact summary vs full list) and the recommended calling sequence. It does not detail error handling, sorting, or pagination, but for a read-only listing tool, this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, then systematically explains modes, usage order, and the parameter. It is reasonably concise, though the Args section partially repeats the earlier 'With repo_name' explanation, causing minor redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple parameter set, absence of annotations, and presence of an output schema, the description covers the essential behaviors and usage sequence. It does not specify outcomes for invalid repos or list sorting, but these details are not critical for effective invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the parameter name and default with no description (0% coverage). The description compensates by explaining that repo_name is optional and that passing it retrieves the full module list for that repository, adding meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'List scanned modules in the knowledge base,' which precisely states the action and target. It further distinguishes the two modes (without repo_name vs. with repo_name) and the context of 'scanned modules' differentiates it from siblings like get_module_facts or search_codebase.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit workflow guidance is provided: 'Always call without repo_name first to discover available repos, then call again with repo_name.' This clearly indicates when to use each mode, but it does not mention alternatives or when not to use the tool, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_module_factsA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the burden of behavioral disclosure. 'Get' implies a read operation, but it does not explicitly state read-only, potential side effects, or prerequisites beyond module discovery. The mention of 'extracted facts' suggests data persisted elsewhere but lacks clarity on behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the core purpose, and every sentence contributes. The argument documentation is integrated clearly without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter read tool with an output schema, the description covers purpose, usage workflow, and parameter format. It omits potential errors or return value specifics, but the output schema likely fills that gap, making it sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description compensates by explaining the module_id format with examples like 'SCAN:backend/app/api/simplified,' adding meaningful context beyond the bare string type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get all extracted facts for a specific module.' It also differentiates from search_codebase by positioning this as a 'deep dive' after discovery.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Use search_codebase first to discover module IDs, then use this for a deep dive.' This directly tells the agent when to use this tool and references an alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_repo_dependenciesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It discloses re data source ('from package manifests, not code analysis'), default behavior ('Without this, returns entire graph'), and the meaning of direction options. This provides useful context, though it doesn't mention error handling or pagination.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a clear purpose sentence, a usage sentence, a data source note, and then an Args section. Every sentence adds value without unnecessary length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists and the tool has only two optional params, the description covers all necessary aspects: what it does, when to use, data source, parameter semantics, and defaults. It is complete for an AI agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It fully explains both parameters: repo_name as a filter with default behavior, and direction with its valid values and their meanings. This goes beyond the schema by clarifying semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Get' and resource 'cross-repo dependency graph' followed by a clear explanation of what it shows. The mention of package manifests (go.mod, package.json) and blast radius distinguishes it from sibling tools like get_module_facts and list_modules.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to understand the blast radius of changes across repositories,' providing a clear context. It also notes that data comes from manifests rather than code analysis, which implies a limitation but doesn't name an alternative tool, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codebaseA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool returns pre-extracted facts ranked by relevance, supports multi-query with semicolons, deduplicates results, and has a max result cap (limit × queries, max 65). It also implies read-only behavior through 'search' and provides cost-saving context, all of which goes beyond the minimal schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a bit lengthy but well-structured into purpose, multi-query instructions, and an args breakdown. Every sentence adds value—examples, defaults, and batching logic are all useful. It's not overly verbose given the tool's complexity, but could be slightly tighter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters, no annotations, and an output schema, the description is remarkably complete. It defines all parameters, gives usage guidance, explains multi-query and limits, and mentions return behavior (facts ranked by relevance). The presence of an output schema means return structure need not be spelled out.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates fully with an 'Args' section that explains every parameter: query, category (with enum-like options), min_confidence (with range and default), limit (with behavior for batched queries), and repo_name. It also gives concrete examples, making parameter usage unambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: 'Search the codebase knowledge base for facts about what the code does, how it's built, and what it assumes.' It also distinguishes itself as the first step before reading source files, setting it apart from the more specialized sibling tools like get_module_facts and get_recent_changes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs 'Use this FIRST before reading source files' and provides strong guidance to 'ALWAYS batch related searches into one call — this is dramatically cheaper.' While it doesn't explicitly name alternatives, the clear directive to use it first and batch searches gives solid usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.