Server Quality Checklist

Tool Scores

• rag_top_serversC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must disclose behavior. It only states 'get top recommended' but does not confirm read-only, mention side effects, permissions, rate limits, or return format. 'Smart defaults' is vague and unhelpful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, short sentence (8 words), which is efficient but lacks front-loaded key details. It communicates the basic purpose but omits critical information, making it borderline under-specified rather than concisely informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 8 optional parameters, no output schema, and no annotations, the agent needs extensive context. The description provides none about output format, parameter roles, or expected behavior. It fails to equip an agent for reliable invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, yet the description provides no parameter explanations. The 8 parameters (categories, minScore, etc.) are completely undocumented in the description. 'Smart defaults' does not clarify parameter purpose or interaction.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description includes a clear verb ('Get') and resource ('top recommended retriever MCP servers'), indicating a retrieval operation. However, it does not differentiate from sibling tools like rag_find_servers, which also retrieves servers. The mention of 'smart defaults' hints at automated filtering but lacks specificity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives such as rag_find_servers. There are no exclusions, prerequisites, or context for optimal usage. The description is insufficient for choosing between tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_find_serversC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description should disclose behavioral traits, but it only says 'search/filter' and omits details like default behavior, ordering, or pagination, making it insufficient for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, but given the tool's complexity (12 parameters), it may be too terse. It is front-loaded but lacks necessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fails to provide essential context for a tool with many optional parameters, no output schema, and no annotations. The agent lacks information on parameter relationships or result structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description does not explain any of the 12 parameters. The agent cannot infer how to use filters like 'query', 'categories', or 'minScore' from the description alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (Search/filter) and the resource (RAG-related MCP servers from the RAGMap subregistry), effectively distinguishing it from siblings like rag_top_servers and rag_get_server.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor does it mention prerequisites or exclusions, leaving the agent without enough context to decide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_get_install_configC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description does not mention side effects, read-only nature, authorization needs, or what the config contains, leaving the agent uninformed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise but sacrifices necessary detail; it could be restructured to include key points without extra length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema or annotations, the description fails to explain what the config includes or its format, leaving the agent to guess about the return value.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single 'name' parameter is not described in the schema (0% coverage) and the description only implies it refers to the server name without further specifics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'install config for a server', distinguishing it from sibling tools like rag_find_servers or rag_explain_score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives; lacks context about prerequisites or typical use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_explain_scoreB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing side effects and constraints. It characterizes the tool as 'Explain', implying a read-only operation, but does not confirm safety, mention any required permissions, or specify behavior on invalid input (e.g., missing server). The description lacks behavioral depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely brief at two sentences, with no superfluous words. The first sentence states the purpose, and the second adds a practical usage hint. Every word earns its place, making it ideal for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description is still incomplete. It fails to describe what the tool returns (e.g., a score breakdown, textual explanation) or how errors are handled (e.g., server not found). Users would need to guess the output format or rely on external documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter 'name' is described only by schema (type string, minLength 1). The description adds an example ('io.github.khalidsaidi/ragmap'), which clarifies the expected format but does not cover other possible naming conventions or validation rules. Schema description coverage is 0%, so the example provides helpful but incomplete guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the verb 'Explain' and the resource 'RAGMap scoring for a server', making the purpose clear. It also provides an example of the expected registry server name. However, it does not elaborate on what the 'explanation' entails (e.g., breakdown of scoring factors), which slightly reduces specificity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description suggests using a registry server name and gives an example, but it does not compare this tool to its siblings (e.g., rag_get_server, rag_find_servers). No explicit guidance on when to use this tool versus alternatives is provided, leaving the agent to infer from the tool's purpose alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_list_categoriesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must bear the full burden of explaining behavior. It states a read-only listing operation, which is appropriate, but omits details about output format or potential side effects, though none are expected.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that directly states the tool's purpose. Every word earns its place; no wasted content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of output schema and annotations, the description could be more complete by specifying the output format (e.g., list of strings or objects). However, for a simple listing tool, the current description is adequate but leaves room for ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the baseline is 4 per rubric. The description does not add parameter information, but none is needed. Schema coverage is trivially 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and the target resource ('RAG categories'), making the tool's purpose unmistakable. It implicitly distinguishes from siblings by focusing on a different entity (categories vs. scores, servers, etc.).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidelines on when to use this tool versus alternatives. Usage is implied by the title and description, but there is no context about prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_get_serverA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description implies read-only but doesn't explicitly state behavioral traits like auth or side effects. Adequate for a simple fetch.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no fluff: first defines action, second gives usage guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers basics for a simple one-param tool with no output schema, but lacks mention of return format or fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds meaning beyond schema by specifying 'registry name' and giving an example, compensating for 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it fetches a server record by registry name (latest version), distinguishing it from sibling tools like rag_find_servers or rag_top_servers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit direction to use exact registry name with an example, but lacks explicit when-not-to-use or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.