Server Quality Checklist

Tool Scores

• index_local_projectC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the action ('index') but lacks critical behavioral details: it doesn't specify if this is a one-time or incremental operation, what happens if the project already exists (overwrite? error?), permission requirements, or any side effects like data persistence or performance impact. This is inadequate for a mutation tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's front-loaded with the core action and target, making it easy to understand at a glance. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of indexing a directory (a mutation operation with potential side effects), no annotations, and no output schema, the description is incomplete. It fails to address key contextual aspects like what 'indexing' entails (e.g., file parsing, embedding generation), error handling, or what the tool returns upon success/failure. This leaves significant gaps for an AI agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters thoroughly. The description adds no additional meaning beyond what's in the schema—it doesn't explain parameter interactions, default behaviors, or usage examples. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('index') and target ('a local project directory into the vector database'), providing a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'list_indexed_projects' or 'search_codebase', which are related but serve different purposes (listing vs. indexing vs. searching).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing directory), exclusions (e.g., when not to index), or how it relates to siblings like 'list_indexed_projects' for checking existing indexes or 'search_codebase' for querying after indexing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codebaseC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool performs a 'semantic search' but doesn't explain what that entails (e.g., natural language understanding, relevance scoring), nor does it cover aspects like rate limits, authentication needs, or whether it's read-only (implied but not explicit). This leaves significant gaps for an agent to understand operational behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without unnecessary words. It is appropriately sized and front-loaded, making it easy for an agent to quickly grasp the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (semantic search with multiple parameters) and the lack of annotations and output schema, the description is insufficient. It doesn't explain what the search returns (e.g., code snippets, file paths), how results are ranked, or any limitations (e.g., indexing requirements). This leaves critical context gaps for effective tool use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with clear documentation for 'query', 'limit', and 'project_filter'. The description adds no additional parameter semantics beyond what's in the schema, such as query format examples or filter usage details. This meets the baseline of 3, as the schema adequately handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('search') and resource ('indexed codebase'), and it specifies the search method ('semantic search'). However, it doesn't explicitly differentiate from sibling tools like 'list_indexed_projects' or 'get_embedding_provider_info', which reduces its score from a perfect 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to prefer 'search_codebase' over 'list_indexed_projects' for browsing projects or 'get_embedding_provider_info' for understanding search capabilities, nor does it specify prerequisites like needing an indexed codebase first.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_embedding_provider_infoB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only states what the tool does without revealing any traits like whether it's read-only, if it requires authentication, what data it returns (e.g., provider name, configuration details), or potential errors. This leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that directly states the tool's purpose without any unnecessary words or fluff. It's front-loaded with the key action ('Get information'), making it easy to parse quickly. Every word earns its place, resulting in an efficient and well-structured description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a tool that retrieves provider information, the description is incomplete. With no annotations and no output schema, it fails to explain what information is returned (e.g., provider type, settings, status) or any behavioral aspects like error handling. This makes it inadequate for users to fully understand the tool's functionality and output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the schema description coverage is 100%, so there are no parameters to document. The description doesn't need to add parameter semantics, and it appropriately avoids mentioning any. This meets the baseline for tools with no parameters, as it doesn't mislead or omit necessary information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('information about the current embedding provider'), making it easy to understand what it does. However, it doesn't differentiate from sibling tools like 'list_indexed_projects' or 'search_codebase', which are unrelated to embedding providers, so it doesn't fully distinguish itself in context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention any prerequisites, such as needing an embedding provider to be configured, or specify use cases like checking provider settings before performing operations. Without this context, users might struggle to determine its appropriate application.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_indexed_projectsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states it's a list operation, implying read-only behavior, but doesn't specify details like pagination, rate limits, or what 'indexed' means in practice. This leaves gaps in understanding how the tool behaves beyond basic listing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words, making it highly concise and front-loaded. It efficiently conveys the core purpose without unnecessary elaboration, which is ideal for this simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters, no annotations, and no output schema, the description is minimally adequate by stating what it does. However, it lacks context on behavior (e.g., return format, limitations) and usage relative to siblings, making it incomplete for optimal agent guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description doesn't add parameter details, which is appropriate, earning a high score as it doesn't introduce confusion or redundancy.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('projects currently indexed'), making the purpose specific and understandable. However, it doesn't differentiate from sibling tools like 'search_codebase', which might also list projects in some context, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, such as 'search_codebase' for filtered searches or 'index_local_project' for adding projects. There's no mention of prerequisites, exclusions, or context for usage, leaving the agent without clear direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.