Server Quality Checklist

Tool Scores

• ingest_documentsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing side effects. 'Ingest' implies a write/mutation, but the description does not mention persistence, collection creation behavior, duplicate handling, chunking consequences, or failure behavior. It does add that it returns chunks added and total, which is useful but insufficient for an unannotated write operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two tight sentences with no fluff, front-loading the core purpose and then giving input guidance. It earns its place, though it is concise partly because it omits important parameter and behavior details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters, zero annotations, zero schema description coverage, and no output schema, this description is not complete enough. It covers the two input sources but leaves chunking parameters and collection behavior to inference, and it only vaguely describes the return value.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains paths ('files/dirs of .txt/.md') and texts ('raw strings'), but it does not explain chunkSize, chunkOverlap, or the collection parameter beyond the vague phrase 'named collection.' This is a significant gap for a 5-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Ingest documents into a named collection so they can be searched.' It clearly distinguishes this ingestion tool from the search/answer/evaluation siblings, and it names the two relevant input forms, paths and texts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies clear usage context: use this tool to add documents to a collection before searching them. However, it never explicitly names alternatives or states when not to use it, so it stops short of the strongest possible routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It does reveal return shape ('each with its source and per-stage scores') and the important empty-result semantics. However, it does not explicitly state that this is read-only, what error conditions exist, or any rate-limit/auth considerations, leaving some behavioral traits implicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler: purpose and output are front-loaded, mode options are compactly summarized, and the empty-result note earns its place as an operational signal. Highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, output, mode options, and empty-result interpretation, which is adequate for a basic call with defaults. Yet with no annotations and no output schema, it leaves the meaning of 'collection' and 'topK' implicit, and it does not mention error behavior or side effects. Serviceable but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it only restates the 'mode' enum and default already present in the schema. It does not explain the semantics of 'query', 'collection', or 'topK' beyond what an agent might infer from names and defaults. This is a real gap for a tool with four parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Search'), a resource ('a collection'), and a concrete output ('the most relevant chunks, each with its source and per-stage scores'). This clearly distinguishes it from sibling tools like answer, ingest_documents, list_collections, and evaluate_retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the primary use case: retrieving relevant chunks from a collection. It does not explicitly name alternatives or state when not to use it, but the context is clear enough that an agent should know to call this for raw retrieval rather than generation or ingestion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• evaluate_retrievalA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description carries the behavioral burden. It conveys that the tool is observational/read-only through 'measure' and 'quantify,' and clarifies it consumes labeled cases rather than modifying documents. It does not spell out side-effect absence or output shape, but the evaluation framing is non-mutating and sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences front-load the action and metrics, then add the case format and purpose. No filler or repetition of schema defaults.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for choosing the tool and constructing the required cases, and it names the metrics returned. However, with no output schema and no annotations, it could usefully state how mode/topK/collection affect the evaluation and that it requires a populated collection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it only explains the required 'cases' parameter ('Each case is {query, relevantSources}'). It adds no meaning for mode, topK, or collection, leaving their semantics to be guessed from names and defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Measure') and a precise resource ('retrieval quality on labeled cases'), then names concrete metrics: hitRate, mrr, recallAtK. This clearly separates it from sibling tools like search (retrieval) and answer (generation).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Use it to quantify quality and catch regressions,' giving a clear context for invocation. It does not name when-not-to-use alternatives, so it falls just short of the highest bar.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• answerA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it does well: it discloses that the tool delegates to the host model via MCP sampling, restricts the answer to retrieved passages only, includes citations, and returns grounded=false when nothing relevant is found. It lacks details about response shape or error behavior, but the core behavioral traits are clearly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with the primary purpose, and every sentence adds meaningful behavioral information. No filler or repetition of schema content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool of moderate complexity with no annotations and no output schema, the description covers the main invocation-relevant behaviors: retrieval, generation, citation, and refusal. It stops short of detailing the exact return structure of grounded=false or citation formatting, but an agent can invoke the tool correctly with the information provided.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter meaning. It implicitly relates 'question' to query and 'collection' to the collection parameter, and 'retrieves relevant passages' hints at topK, but it never explicitly explains topK or the meaning of the defaults. The description adds some context but not enough to fully clarify all three parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a clear verb-resource pair ('Answer a question grounded in a collection') and adds differentiators: citations, retrieval, MCP sampling, and refusal. This clearly separates it from siblings like search or list_collections.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly implies when to use the tool: when the agent needs a grounded, cited answer synthesized from relevant passages in a collection. It does not explicitly name alternatives or exclusions like 'use search for raw passages', but the contextual phrasing is strong enough that an agent can infer the appropriate use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_collectionsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral transparency burden. It clearly conveys that the operation is a non-mutating listing action and discloses what the agent will receive: collection names and chunk counts. As a zero-parameter listing tool, this is sufficient transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no wasted words. It front-loads the action ('List all ingested collections') and then supplies the key output detail, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, but the description tells the agent exactly what the tool returns. It names the resource, the scope ('all ingested'), and the relevant output field (chunk count), so nothing critical is missing for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description does not need to explain parameter meaning because the schema is empty and there is nothing to document.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'List all ingested collections' and adds the chunk-count detail, making the tool's purpose unmistakable. It is clearly distinct from the sibling tools, which handle ingestion, search, answering, and retrieval evaluation rather than listing collections.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: use this tool when you need the set of ingested collections and their chunk counts. It does not explicitly name alternatives or exclusions, but none of the sibling tools appear to overlap with this listing function.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.