Server Quality Checklist

Tool Scores

• list_sourcesB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It only states the tool lists documents and chunks, without mentioning side effects, authentication needs, rate limits, or whether it's read-only. Minimal transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single concise sentence that front-loads the purpose. No unnecessary words, every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and an output schema exists, the description is mostly complete. It could mention scope (e.g., all indexed documents) or output sorting, but the basic functionality is clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist in the schema, so baseline 4 applies. The description adds no parameter info, but with zero parameters, it is adequate. Schema coverage is trivially 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists documents and their chunk counts, distinguishing it from siblings like index_path, search, and stats. The verb 'List' and resource 'documents currently indexed' are specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like stats or search. The description does not mention any context, prerequisites, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• index_pathA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It describes the indexing process and return value, but does not mention that this is a write operation that may be resource-intensive, require authentication, or have potential side effects like modifying the vector store. The description is incomplete for behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the action and supported types, followed by the process and return summary. Every sentence is informative and there is no extraneous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, and the description adequately explains the return value (summary of indexed/skipped counts). However, it lacks details on error handling, permission requirements, or performance implications for large directories. Given the tool's simplicity and the presence of output schema, the description is reasonably complete but could be enhanced.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description has 0% schema coverage, meaning it does not explicitly name or explain the parameters 'path' or 'recursive'. It vaguely implies path via 'file or folder', but does not clarify the default recursive behavior or how to control it. The description adds minimal semantic value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Index' and the resource 'a file or a whole folder of documents'. It lists supported file types (pdf, docx, etc.) and explains the process (extracts text, chunks, embeds, stores vectors). This effectively distinguishes it from sibling tools like 'search' (which queries indexed data) and 'list_sources' (which lists sources).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that this tool is used to add documents to the index before searching, but it does not explicitly state when to use it versus alternatives, nor does it mention prerequisites or limitations. No guidance is given on when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It implies a read-only operation via 'Show,' but does not explicitly guarantee no side effects or mention any behavioral traits like speed or auth requirements. Adequate for a simple status tool but could be more explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of nine words, front-loaded with the action and resource. No wasted words; every part adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and an output schema exists (not shown but referenced), the description is complete enough. It lists the key output elements and is sufficient for this simple status tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema description coverage is 100% trivially. The description does not need to add parameter semantics, so baseline 4 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool shows retriever status and lists specific components (embedding backend/model, storage location, counts). It is a specific verb+resource and clearly distinguishes from siblings like search or list_sources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives (e.g., search, list_sources). The description only states what it does without context on prerequisites or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description fully compensates by disclosing that results are chunks with scores and that the tool itself does not answer. No contradictions present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first for function and output, second for usage advice. No redundancy, front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, no annotations), the description covers purpose, output, and usage pattern thoroughly. No missing critical details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description explains both parameters: 'query' as the search term and 'k' as the top number of chunks with a default of 5. Could add constraints like allowed range for k, but sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Search' and the resource 'indexed documents', listing the returned fields (source path, similarity score). It distinguishes from sibling tools like index_path and list_sources by focusing on retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to use the passages as grounding and warns that the tool does not answer, guiding the agent on proper usage. However, it lacks contrasting with alternatives for when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.