Server Quality Checklist

Tool Scores

• rag_healthA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility. It discloses the endpoint and the intent, but fails to mention what a successful check looks like (e.g., return value, HTTP status, or error behavior), leaving ambiguity for the agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence that directly states the purpose and URL. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple health check with no parameters and no output schema, the description gives the endpoint and purpose but omits how results are returned or interpreted, which the agent needs to act on the outcome.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the baseline of 4 applies. The description appropriately focuses on the action and target endpoint without needing parameter detail.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Identifies a specific action ('Check whether') targeting a specific resource (the dryad-rag-pipeline service at the given URL). This is clearly distinct from sibling tools rag_ingest and rag_query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for verifying service reachability but does not explicitly state when to use it versus alternatives, mention exclusions, or provide context like 'run before other rag tools'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_queryA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Given no annotations, the description carries the full burden. It discloses what the tool returns (chunks, source file, similarity score) and its purpose (citation). However, it doesn't explicitly state read-only behavior, potential side effects, or limitations (e.g., no matching results, index readiness). It's adequate but leaves some behavioral traits undisclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and front-loaded with the primary action. It contains no filler or redundant information. Every clause adds value: action, scope, output, and purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple query tool with two parameters and no output schema, the description is complete enough. It explains the return payload (chunks with source and score) and the intended use (citation). It doesn't address edge cases like empty results, but that's not critical given the tool's simplicity and the high schema coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage for both parameters (question and top_k), so the description doesn't need to repeat them. The description adds no parameter-specific meaning beyond the schema, but the baseline of 3 applies because the schema already documents the parameters thoroughly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Ask a question against the indexed documents and get back the most relevant chunks with their source file and a similarity score, for citation.' It uses a specific verb ('ask') and describes the resource (indexed documents) and output (chunks, source file, similarity score). This distinguishes it from sibling tools rag_health and rag_ingest, which clearly serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: it is the tool for querying the RAG index with a natural language question. While it doesn't explicitly exclude alternatives or state when-not-to-use, the clear action ('Ask a question') and context ('against the indexed documents') provide sufficient guidance. It doesn't mention rag_ingest or rag_health, but the different verbs in the sibling names make the distinction obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_ingestA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the core operation ('re-index'), but does not mention whether it clears the existing index, whether it is idempotent, or whether it may cause downtime or delays. While 're-index' implies rebuilding, important behavioral details are omitted, making 3 the appropriate score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, both necessary. The first states the action and resource; the second provides usage guidance. No fluff, front-loaded, and perfectly sized for a tool with no parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with no parameters, no output schema, and clear sibling relationships, the description fully covers the necessary information: what it does, when to call it, and its role relative to rag_query. There is no ambiguity left for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so parameter semantics is largely moot. The description adds context by mentioning the 'configured markdown document set', implying the source is externally configured. This provides useful meaning beyond the empty schema, justifying a baseline 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb 'Re-index' and clearly identifies the resource ('configured markdown document set on the RAG service'). It distinguishes from siblings by explicitly referencing the precondition for rag_query and the action of re-indexing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use instruction: 'Call this once before rag_query if the index is empty or the docs changed.' This directly names the sibling tool and gives concrete conditions, making it easy for an agent to decide when to invoke this tool instead of rag_query.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.