Server Quality Checklist

Tool Scores

• assert_generateA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses key behavioral traits: it generates test scenarios, can optionally save them to Assert, and defaults to returning a markdown preview. However, it lacks details on permissions needed, rate limits, error handling, or what 'ready-to-run' entails (e.g., format specifics, dependencies). This is adequate but leaves gaps for a mutation-capable tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by an optional feature in the second. Both sentences earn their place by clarifying functionality and user choice. It is appropriately sized with zero waste or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, 4 parameters with 100% schema coverage, and no output schema, the description is moderately complete. It covers the tool's purpose and key behavior (generation and optional saving), but lacks details on output format (beyond 'markdown preview'), error cases, or integration context. For a tool that creates test scenarios, more guidance on output expectations would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds marginal value by implying the 'description' parameter is for plain-English input and 'save' controls persistence, but does not provide additional syntax, format, or usage details beyond what the schema states. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Generate a ready-to-run E2E test scenario'), the resource ('in Assert Markdown format'), and the transformation ('from a plain-English description'). It distinguishes from sibling tools like assert_list, assert_run, and assert_status by focusing on test creation rather than listing, executing, or checking status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: when you need to create E2E test scenarios from English descriptions. However, it does not explicitly state when not to use it or name alternatives (e.g., use assert_run for execution, assert_list for browsing). The optional save parameter implies a choice between preview and persistence, but no explicit guidance on choosing between this and other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assert_statusA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: it's a polling operation (implies repeated calls may be needed), returns pass/fail status, provides actionable failure details, and includes screenshot URLs for failed steps. It doesn't mention rate limits, authentication needs, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with zero waste. First sentence states the action and resource, second sentence details the return values. Every word earns its place and information is front-loaded appropriately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no output schema, the description provides good context about what the tool returns (pass/fail status, failure details, screenshot URLs). It could be more complete by specifying the polling interval or whether this is a blocking call, but covers the essential behavior well given the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (run_id is fully documented in schema), so baseline is 3. The description doesn't add any parameter-specific information beyond what the schema already provides about the run_id parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('poll', 'returns') and resources ('test run', 'status', 'step-level results'). It distinguishes from siblings by focusing on status checking rather than generating, listing, or initiating runs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after a test run has been initiated (since it requires a run_id from assert_run), but doesn't explicitly state when to use this tool versus alternatives. No explicit guidance on when-not-to-use or direct comparison to siblings is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assert_listA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'Supports filtering by project and pagination,' which adds useful context beyond basic listing. However, it lacks details on permissions, rate limits, or response format, leaving gaps for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and usage context. Every sentence earns its place by providing essential information without redundancy, making it highly efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description does well by covering purpose, usage guidelines, and behavioral hints like filtering and pagination. However, it lacks details on return values or error handling, which could be useful for a list tool with no structured output information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description adds marginal value by mentioning filtering by project and pagination, but does not provide additional syntax or format details beyond what the schema specifies, aligning with the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and resource 'existing E2E test scenarios saved in Assert,' making the purpose specific and unambiguous. It distinguishes from siblings like 'assert_generate' (create), 'assert_run' (execute), and 'assert_status' (check status) by focusing on retrieval of saved test scenarios.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool: 'to audit coverage before generating new tests.' This provides clear context for usage and distinguishes it from alternatives like 'assert_generate' for creating tests, offering practical guidance for the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assert_runA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: the operation is async (not immediate), returns a run ID for tracking, and requires polling with assert_status for completion. It doesn't mention error handling, timeout behavior, or authentication requirements, but covers the essential execution model.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with three sentences that each earn their place: states the purpose, explains the input options, and provides critical usage guidance about async behavior. No wasted words, front-loaded with core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (async execution with polling), no annotations, and no output schema, the description does well by explaining the execution model and relationship to assert_status. It could be more complete by mentioning what the run ID represents or error cases, but covers the essential context for proper tool selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema by mentioning the scenario_id/markdown alternatives and the async nature, but doesn't provide additional semantic context about parameter interactions or usage patterns.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Execute a test scenario'), the resource ('test scenario'), and the outcome ('return a run ID'). It distinguishes from siblings by specifying this tool initiates runs while assert_status polls for completion and assert_list/assert_generate handle other operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides when-to-use guidance: use this tool to start a test run, and use assert_status to poll for completion. It also distinguishes between using saved scenarios (scenario_id) vs. ad-hoc markdown, though it doesn't explicitly mention when to choose between these alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.