Server Quality Checklist

Tool Scores

• searchB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Only states it returns ranked results, but omits details like pagination, handling of empty queries, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences with no redundant information; front-loaded with the key action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 4-parameter tool with output schema, description is too minimal. Does not explain tags filtering, limit usage, or detail levels, leaving significant gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50%; description adds no meaning beyond 'query'. Parameters 'tags' and 'limit' lack both schema descriptions and description-based clarification.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search for available tools by query' with a verb and resource. It distinguishes from siblings like tags, get_schema, and execute by being the primary search tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives. Does not mention contexts, exclusions, or when-not-to-use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_schemaA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description does not disclose any behavioral traits such as side effects, authorization needs, or rate limits. For a read-only tool, the description could mention that it is non-destructive, but it remains silent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. Essential information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool is simple, output schema exists, so description doesn't need to explain return values. The description covers the essential usage flow, but slight improvement could mention that it returns JSON schemas.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds context about using after searching, which helps understand the purpose of the 'tools' parameter, but doesn't add extra meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool gets parameter schemas for specific tools. Verb 'get' and resource 'parameter schemas for specific tools' are specific and distinct from sibling tools like execute, search, and tags.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'Use after searching to get the detail needed to call a tool.' This gives clear context for when to use it, though it doesn't mention when not to use or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tagsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite lacking annotations, the description accurately conveys a read-only listing behavior. No side effects or additional traits needed; the simplicity of the tool makes the description sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two short sentences, front-loading the main purpose immediately. Every word is purposeful without any waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, one optional parameter, and an existing output schema, the description is nearly complete. It could mention that results include tag names and counts or tools, but the output schema fills this gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides full coverage (100%) with a clear description of the 'detail' parameter including enum values. The description adds no extra parameter details beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists available tool tags, with a specific verb and resource. It distinguishes from siblings like 'search' and 'execute' by focusing on browsing tags before searching.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description suggests using 'tags' to browse available tools by tag before searching, providing a clear when-to-use hint. However, it does not explicitly mention when not to use or compare to alternatives like 'search'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• executeA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and delivers detailed behavioral information: sandbox guardrails (no network, no filesystem, blocked imports), missing builtins, error behavior, and session-scope nuances. It is exceptionally transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very long but well-structured with clear sections (sandbox guardrails, builtins, auth, session-scope contract). While length is justified by complexity, it could be more concise without losing essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and lack of output schema, the description covers all necessary aspects: purpose, usage, environment, error handling, auth, and session management. It leaves no critical gaps for the AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although schema description coverage is 100%, the description adds immense value beyond the schema's brief parameter description. It explains the execution environment, available functions (call_tool), and how to structure code, vastly enriching parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear statement: 'Run async Python in a sandboxed interpreter.' It explicitly describes the tool's function and distinguishes it from siblings (tags, search, get_schema) which are unrelated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides extensive guidance including session-scope contract, auth handling, error semantics, and explicit do's and don'ts (e.g., 'Call get_session_state first', 'Do NOT rapid-poll inside a single execute block'). It effectively tells the agent when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.