Server Quality Checklist

Tool Scores

• test_caseA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (destructiveHint=true), the description explicitly warns that delete is destructive and requires confirmation, clarifies that create does not auto-run the test, and mentions soft-delete. This adds valuable behavioral context, though it does not cover rate limits or authorization.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured: a one-sentence overview followed by clear action-specific bullet points. Every sentence adds value, with no unnecessary content. The front-loaded purpose immediately orients the agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 12 parameters, 3 actions, and no output schema, the description covers the parameter requirements well but omits return values for create, update, and delete. The absence of output schema documentation is a gap, though annotations partially compensate for behavioral hints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description groups parameters by action, showing required combinations (e.g., suiteUuid or suiteName+project for create). While the schema already uses action-prefixed descriptions, the description's structured grouping significantly aids understanding. Baseline 3 due to 100% schema coverage, but extra value raised score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool manages test cases within a suite and enumerates three actions (create, update, delete) with specific parameter requirements. It distinguishes itself from sibling tools like test_suite and executions by focusing on individual test case operations and noting that create does not auto-run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implicit usage guidance through action-specific parameter lists (e.g., suiteUuid vs. suiteName+project for create), but does not explicitly state when to use this tool over alternatives like test_suite or executions. No direct comparison to siblings is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• test_suiteA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations mark destructiveHint=true, and the description adds context by noting that the 'delete' action is a soft-delete that requires confirmation. It also discloses that 'run' executes tests asynchronously and suggests polling with the 'results' action. While the description does not detail all edge cases, it sufficiently complements the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with a clear introductory sentence followed by a bulleted list of actions. It is concise for the complexity (12 parameters) but could be slightly tighter by removing redundant phrasing (e.g., repeated identification pattern). Overall, it is well-organized and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema and the tool's complexity (multiple actions, async behavior), the description covers identification and action parameters well but does not describe return values (e.g., paginated list format, test outcomes structure). Agents would need additional inference for expected outputs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of parameters with descriptions, but the description adds value by grouping parameters per action (e.g., [list] parameters, [create] parameters) and explaining composite identifiers like 'suiteName + a project identifier'. This reduces ambiguity beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Manage and run test suites' and enumerates five specific actions (list, create, run, results, delete), each with distinct purposes. This distinguishes the tool from siblings like 'test_case' and 'executions' by focusing on test suite lifecycle management.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on how to identify a suite (by UUID or name+project) and details each action's parameters. However, it does not explicitly contrast with sibling tools (e.g., when to use 'test_case' instead of 'test_suite') or state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• executionsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=true and openWorldHint=true, and the description does not contradict them. It adds behavioral details: 'get' returns full detail including nodeExecutions, state, errorInfo, and artifact URLs; it advises polling until artifact URLs are available. This enhances transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: two clear sentences plus a short tip. It is front-loaded with the main purpose, then dives into details. Every sentence provides useful information, with no redundancy or excessive length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the comprehensive schema, annotations, and lack of output schema, the description covers essential aspects: two actions with their parameters, and a usage tip. It does not explain pagination further, but the schema already describes page and pageSize. Overall, it is adequate for an agent to correctly invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds value by grouping parameters by action (e.g., '[list]' and '[get]' prefixes) in the schema descriptions, and in the free text it clarifies which parameters apply to each action. This improves understanding beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Look up workflow executions' and lists the types of executions (check_app_in_browser, trigger_crawl, test-suite runs). It differentiates between 'get' and 'list' actions, providing specific verb+resource semantics. No confusion with sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear guidance on when to use 'get' vs 'list' by specifying the parameters for each action. It also includes a practical tip about polling after a fresh check_app_in_browser run. However, it does not explicitly state when not to use this tool or compare it to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_app_in_browserA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial behavior beyond the annotations (readOnlyHint:false, openWorldHint:true, destructiveHint:false): automatic localhost tunneling, the ~25-step agent budget and timeout risk, warm-session reuse vs freshSession forced real login, and the credential fallback behavior ('it falls back to the environment's stored credential'). It even discloses what the result reports (`logins` identity). This is rich, accurate transparency with no contradiction against annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but purpose-built for a genuinely complex tool (12 params, nested auth objects, credential governance). It is sensibly front-loaded: purpose, then localhost support, then scoping, then credentials. There is some redundancy between the credentials prose and the schema-level descriptions of auth/loginCredentials/useEnvironmentCredentials, but the prose earns its place by adding precedence semantics the schema lacks.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex, credential-heavy tool with no output schema, the description covers the hard parts thoroughly: what the tool does, localhost tunneling, per-call scoping with a step budget, and the full matrix of credential mechanisms and their precedence. It even notes the output identity field (`logins`), compensating for the absent output schema. Nothing an agent needs to invoke it correctly is left unexplained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds real semantic value above the schema by explaining relationships and precedence among the credential parameters (username/password vs auth.username vs loginCredentials vs credentialId vs useEnvironmentCredentials), including when each applies (part-way login, precondition login, forbidding auto-fill). This inter-parameter guidance is not derivable from the schema text alone, justifying a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific purpose with concrete verbs and resource: 'Give an AI agent eyes on a live website or app... browses it, interacts with it, and tells you whether a given task or check passed.' It names clear use cases (visual QA, flow validation, regression checks) and qualifies itself as 'anything that needs a real browser to verify,' which helps separate it from sibling browser tools like probe_page and trigger_crawl. This goes well beyond a tautology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use context ('Use for visual QA, flow validation, regression checks, or anything that needs a real browser to verify') and unusually strong how-to guidance: one focused check per call, a ~25-step internal budget, advice to split multi-step flows into separate calls with a concrete three-call example. It lacks explicit exclusions and does not name sibling tools as alternatives, but the usage context is clear enough to be a solid 4.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• environmentA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (destructiveHint, readOnlyHint), the description discloses critical non-obvious behaviors: passwords are never returned ('passwords never returned'), credentials are write-only, delete is explicitly labeled DESTRUCTIVE and requires confirmation, projectUuid auto-resolves from the git repo when omitted, and warm session reuse explains surprising check results. It even explains why a check can report 'no login form' when already authenticated. This is rich behavioral disclosure that annotations alone could not provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but efficiently structured: a one-line action index followed by compact signature-like bullets, then a SESSIONS paragraph that explains a non-obvious cross-tool behavior. Every piece has a purpose, and the most important caveats (destructive delete, password concealment) are front-loaded. It is long because the tool has seven actions, but nothing feels redundant or fluffy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's high complexity (7 actions, 16 parameters, no output schema), the description is remarkably complete. It covers every action's inputs, key constraints (confirmation, pagination, auto-resolution, credential management), and the session-reuse context needed to interpret results correctly. It also connects to a sibling tool for an alternative approach. No critical operational detail appears missing for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema coverage is 100%, so the baseline is 3. The description adds meaningful action-level semantics that the flat schema lacks: it groups parameters by action (e.g., 'get {uuid, projectUuid?} → one environment with credentials inline'), clarifies the confirm flag's conditional need, and explains that passwords are write-only and never returned. This goes beyond the schema's per-parameter descriptions, justifying a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Manage environments (and their login credentials) under a project.' It then enumerates seven distinct actions, making the scope unmistakable and distinguishing it from siblings like project, test_suite, and check_app_in_browser. The action list functions as a precise mini-spec that could not be confused with any other tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use each action and explicitly references a sibling for an alternative behavior: 'pass freshSession:true on a single check_app_in_browser call to bypass reuse without clearing anything.' It also implies when sessions/clearSessions should be used. It does not explicitly state 'use this instead of X' for all siblings, but the action breakdown makes the intended use unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• projectA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds value beyond annotations by detailing that 'create' requires the repo to be GitHub-linked and names resolve by case-insensitive exact match. Annotations indicate not read-only and not destructive, which aligns with the create action. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single paragraph with a clear structure: an introductory sentence followed by bullet-style action definitions. It is concise and front-loaded, though a list or subheadings could improve scanability. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema, the description adequately describes return types for each action (full detail vs. paginated summaries). It covers all actions and notes important constraints, but lacks details on error handling or pagination behavior beyond page/pageSize.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage with descriptions, but the description augments this by grouping parameters per action (e.g., (teamUuid|teamName)) and adding constraints like 'repo must be GitHub-linked'. This provides useful context beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it manages DebuggAI projects and enumerates three distinct actions (get, list, create), each with a specific verb and resource. It explicitly notes that update/delete are not available, differentiating from any potential sibling tools for project mutation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear guidance on when to use each action (e.g., 'get' for full detail, 'list' for summaries) and explicitly states that update/delete are not available here, directing to the web app. While it doesn't compare directly to the listed sibling tools, the siblings are sufficiently different (e.g., executions, browser checks) that the usage context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trigger_crawlA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate openWorldHint=true (side effects) and readOnlyHint=false, and the description aligns by detailing that the crawl populates a knowledge graph asynchronously. It adds critical behavioral context: localhost support via automatic tunneling, long-running nature, and that the return value is execution status (not pass/fail). No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (LOCALHOST SUPPORT, WHEN TO USE, SCOPE) and front-loads the primary purpose. It is efficient but slightly verbose in parameter descriptions (e.g., repeating credential guidance). Still, every section serves a purpose and avoids redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (9 parameters, no output schema, no nested objects), the description covers all necessary aspects: purpose, usage context, behavioral traits, and parameter semantics. It explains what the tool returns (executionId+status+outcome) and addresses long-running behavior. No major gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds significant value beyond the schema by explaining localhost handling for 'url', auto-detection for 'projectUuid' and 'repoName', and important caveats for 'username' and 'password' (e.g., 'do NOT invent credentials'). It also clarifies optional overrides for environment and credential. This extra context raises the score to 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool triggers a browser-agent crawl to build a project's knowledge graph. It uses a specific verb ('trigger') and resource ('crawl'), and explicitly distinguishes it from the sibling tool 'check_app_in_browser' by identifying the appropriate use case (post-feature vs. per-change verification).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use (after significant features, new environments, onboarding) and when not to use (per-change verification, recommending check_app_in_browser instead). It also clarifies scope (one crawl per call) and expected duration (minutes to tens of minutes), leaving no ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• probe_pageA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial behavioral context far beyond the annotations: localhost auto-tunneling with fast-fail pre-flight probe, batch-mode session/timeout sharing, the content-settle readiness semantics ('DOM going quiet' rather than network idle or load), and partial-failure behavior where a single failed target's error appears in result.error without failing the whole batch. The readOnlyHint and openWorldHint annotations are consistent with this read-only probing tool, and there is no contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The text is lengthy but deliberately sectioned with labeled headers (WHEN/WONT NOT use, LOCALHOST SUPPORT, BATCH MODE, READINESS), so an agent can scan or read with minimal overhead. Every sentence carries a distinct functional fact; a few points (e.g., the full waitForLoadState/networkidle discussion) could be tightened but are not filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, so the description must carry the return contract - and it does: it lists screenshot, title/finalUrl/statusCode/loadTimeMs, structured console errors, per-URL network summary, and the includeHtml option. Batch limits, localhost handling, timeouts, wait options, and error semantics are all covered; nothing an agent needs to call the tool correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Since schema description coverage is already 100%, the baseline is 3 and the schema does the heavy lifting. The description adds genuine decision-making beyond that baseline: waitForSelector nuance for SPA mounts, the explanation that networkidle is accepted but never issued against live sites, and batch-mode guidance that up to 20 targets share one execution unit for speed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource - 'Probe one or more URLs and return their rendered state' - and enumerates the exact return payload (screenshot, title/finalUrl/statusCode/loadTimeMs, console errors, per-URL network summary). It also distinguishes itself from its sibling by naming check_app_in_browser explicitly in the NOT FOR section.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The WHEN TO USE section gives concrete agent-facing queries ('did I just break /settings?', 'smoke-test these 5 routes') and the decision constraints (fast, no LLM cost, no agent loop). The NOT FOR section explicitly rules out scenario verification and interaction and names the alternative: 'Use check_app_in_browser for those.' An agent knows exactly when to pick this tool over its sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.