Server Quality Checklist

Tool Scores

• check_fileA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries full burden. It states the tool runs locally, but does not explicitly mention read-only nature, permissions needed, or side effects. For a security analysis tool, more transparency would be beneficial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. The first sentence concisely states purpose and threats, the second lists supported files and mentions locality. Well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, threats, and supported files. However, it does not describe the output or result format (e.g., list of threats, scores). With no output schema, this is a notable gap for a security analysis tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a description for the 'path' parameter. The tool description adds value by listing supported file patterns (e.g., .cursorrules, CLAUDE.md), providing context beyond the schema's examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it analyzes AI config files for specific security threats, and lists supported files. It distinguishes from sibling tools like check_environment or check_server which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when dealing with AI config files, but does not explicitly state when to use vs alternatives or when not to use it. Usage guidelines are implied but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_registryA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description should convey behavioral traits. It mentions returns (trust scores, security ratings) and scale (3400+ servers, 265 automated probes), but lacks details on side effects, rate limits, or data freshness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences with no extraneous information: one for purpose and one for usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the description states what the tool returns and when to use it, it omits ordering, filter behavior (e.g., safe_only), and error handling. Without an output schema, more detail on return format would be valuable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds a query example but does not significantly enhance understanding beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('search') and resource ('AgentSeal security registry of 3400+ scanned MCP servers'), clearly distinguishing it from sibling tools like check_server or check_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The explicit statement 'Useful when the user asks to find, discover, or recommend MCP servers' provides clear context for use, though it does not mention when to avoid the tool or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_serverA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It transparently describes that AgentSeal will fetch the server, run 265+ security probes, and add results to the public registry. This adequately discloses the tool's behavioral effects, though it could mention idempotency or duplicate handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main action and followed by a critical requirement. Every word adds value, and there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description lacks details on the return value or error handling. While it covers purpose, behavior, and a key requirement, it omits information about what the agent can expect as a response or how to handle failures.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds minimal parameter insight beyond the schema, only reiterating the 'confirmed' requirement. No additional meaning is provided for 'name' or 'package_type' beyond what the schema already offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool submits an MCP server not in the registry for security scanning, with a specific verb and resource. It distinguishes itself from sibling tools (check_environment, check_file, check_server, search_registry) by focusing on submission, not inspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the tool is for servers not in the registry, giving a clear usage condition. It also highlights the 'confirmed=true' requirement to prevent accidental submissions. However, it does not provide explicit when-not-to-use or compare directly with alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_environmentA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears the full burden and clearly discloses a key behavioral trait: 'Only server names are sent to the API – credentials and config values stay local.' This addresses privacy concerns. It also implies the tool is read- only and non-destructive, though it does not detail auth needs or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, with the first sentence stating the core function and the second adding an important privacy guarantee. Every sentence is meaningful and front-loaded, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description hints at the output ('shows trust scores and flags risky or unknown servers') but does not specify the format or structure. Given the tool has no output schema, additional detail would be beneficial, though the current description is sufficient for basic understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides descriptions for both 'scope' and 'dry_run' parameters (100% coverage). The description adds no additional semantic information beyond what the schema provides, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the action ('scan MCP servers'), the resources ('AI tools: Claude Desktop, Cursor, Windsurf'), and the purpose ('check each against AgentSeal registry, show trust scores'). It distinguishes itself from siblings like check_file and check_server by focusing on the environment's installed servers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states when to use the tool (to check installed servers) and what it does (scans and checks registry). However, it does not explicitly contrast with alternatives like search_registry or check_server, nor does it specify when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_serverA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description does not confirm read-only behavior nor mention side effects. It implies a non-destructive audit via 'returns' and 'probes', but lacks explicit disclaimers about permissions or reversibility.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core purpose, followed by return details and usage hint. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter check tool with no output schema, the description adequately covers purpose, output detail, and usage context. No missing elements.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the parameter description provides concrete examples ('@modelcontextprotocol/server-github', 'filesystem'), adding meaningful context beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the action ('check the security') and the resource ('specific MCP server'). Specifies return values (trust score, risk level, etc.) and distinguishes from sibling tools (check_environment, check_file, etc.) by focusing on server security assessment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use case ('before installing or recommending an MCP server'). However, does not mention when not to use or directly contrast with sibling tools, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.