Server Quality Checklist

Tool Scores

• add_text_fileC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions the tool adds a file and downloads from a URL, but doesn't disclose behavioral traits like error handling (e.g., invalid URLs, duplicate names), permissions needed, or side effects (e.g., overwriting existing files). The constraint about 'docs' is helpful but insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded with the core action, followed by a specific constraint. Both sentences are relevant, with no wasted words, though it could be slightly more structured (e.g., separating action from constraints).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and a mutation tool with two parameters, the description is incomplete. It lacks details on return values, error conditions, or operational context (e.g., how files are stored, success indicators), leaving gaps for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters fully. The description adds no additional meaning beyond the schema's details for 'document_name' and 'file_url', such as format examples or usage tips, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Add a text file') and target location ('to the docs directory'), with a specific naming constraint. It distinguishes from sibling tools like 'add_git_repository' (different resource) and 'list_documents'/'rag_query' (different actions), though it doesn't explicitly contrast them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a constraint ('do not use 'docs' in the document name') but offers no guidance on when to use this tool versus alternatives like 'add_git_repository' for repositories or 'list_documents' for viewing. It lacks context on prerequisites or typical use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_git_repositoryB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions that the tool adds a repository and supports sparse checkout, but does not describe key behaviors such as what happens if the repository already exists, whether it requires authentication, if there are rate limits, or what the output looks like. For a mutation tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, consisting of two sentences that directly state the tool's purpose and a key instruction. There is no wasted language, and every sentence serves a clear purpose in guiding usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (adding a git repository with sparse checkout), lack of annotations, and no output schema, the description is incomplete. It fails to address important contextual aspects such as error handling, success criteria, or what the agent should expect after invocation, leaving gaps for effective tool use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description adds minimal value beyond the schema by implying the tool's purpose relates to these parameters, but does not provide additional syntax, format details, or usage examples that aren't already covered in the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Add a git repository to the docs directory with optional sparse checkout.' It specifies the verb ('add'), resource ('git repository'), and destination ('docs directory'), but does not explicitly differentiate it from sibling tools like 'add_text_file' or 'list_documents' beyond the type of resource being added.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some implied usage guidance by mentioning 'optional sparse checkout' and a specific instruction: 'Please do not use 'docs' in the document name.' However, it does not explicitly state when to use this tool versus alternatives like 'add_text_file' for non-git content or 'rag_query' for querying documents, nor does it outline prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_queryA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It helpfully reveals that 'If the index does not exist, it will be created when you query, which may take some time' - this is valuable behavioral context about performance implications and automatic index creation that isn't obvious from the tool name or schema alone.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences. The first states the core purpose, the second provides important behavioral context. Both sentences earn their place, though the structure could be slightly improved by front-loading the behavioral note more clearly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a query tool with 2 parameters and no output schema, the description provides adequate context about the core operation and an important behavioral note. However, it doesn't describe what the query returns (format, content), error conditions, or authentication requirements, leaving some gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both parameters (document_id and query). The description doesn't add any additional parameter semantics beyond what the schema provides, such as query format examples or document_id constraints. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Query a document using RAG' (verb+resource). It distinguishes from siblings like 'add_git_repository' and 'add_text_file' by focusing on querying rather than adding content. However, it doesn't explicitly differentiate from 'list_documents' in terms of querying vs listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage context with the note about index creation, implying this tool should be used for querying documents with RAG. However, it doesn't explicitly state when to use this vs alternatives like 'list_documents' or provide clear when-not-to-use guidance beyond the performance implication mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_documentsA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It indicates this is a read operation ('List') and suggests it's a preliminary check tool, but doesn't specify behavioral traits like whether it requires authentication, has rate limits, or what format the output takes. It adds some context but lacks comprehensive behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that each serve a distinct purpose: the first states what the tool does, the second provides usage guidance. There's zero wasted language and it's front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no parameters and no output schema, the description provides good context about the tool's purpose and when to use it. However, without annotations or output schema, it could benefit from more information about what the output looks like (e.g., format, structure) to be fully complete for agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema already fully documents the parameter situation. The description appropriately doesn't add parameter information beyond what's in the schema, maintaining a clean baseline. No compensation is needed for missing parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List all available documents') and resource ('in the DOCS_PATH directory'), distinguishing it from sibling tools like add_git_repository or add_text_file which perform different operations. It explicitly identifies what the tool does without being vague or tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Always use this tool first to check if desired documents already exist before adding new ones'), including a clear alternative scenario (using add_git_repository or add_text_file for adding documents). This gives strong context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.