Server Quality Checklist

Tool Scores

• audit_projectA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses caching and output type, but does not mention whether the operation is read-only, auth requirements beyond token, or potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences: first defines purpose with concrete examples, second notes output and caching. No redundant words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Describes output format vaguely ('findings JSON') and mentions caching for apply_fix tools, but lacks detail on return structure (no output schema) and does not explain how list_findings relates to cached results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds minimal value beyond schema descriptions (e.g., token optional with env var is already in schema). No additional parameter guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Explicitly states it scans a Supabase project for security issues and lists specific categories (RLS gaps, etc.), clearly distinguishing from sibling tools that apply fixes or list findings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Mentions caching for use by apply_fix tools, hinting at a workflow, but does not explicitly state when to use versus siblings like list_findings, nor when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• preview_fixA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that the tool does not apply changes (wrapped in rollback) and is safe. With no annotations, it adequately covers the non-mutating behavioral trait.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, no wasted words. Efficiently conveys the core function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequate for a simple preview tool, but lacks explanation of return format or what 'returns what would have happened' means. No output schema provided.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50%, but description adds no meaning for parameters. 'project_ref' lacks description in both schema and description, missing opportunity to explain its role.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool previews changes without applying, wrapping fix SQL in BEGIN...ROLLBACK. It distinguishes from siblings like apply_fix by focusing on preview-only behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states it is safe to call for any finding, implying usage for testing/reviewing. Does not explicitly exclude scenarios or name alternatives, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• apply_all_fixesA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the transactional behavior (wraps in single transaction, rolls back on failure) beyond the absent annotations. This is key behavioral information, though permissions or side effects are not mentioned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences efficiently convey purpose, transaction behavior, and usage advice with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core operation and advice but omits return value details (no output schema) and does not reference sibling tools like 'list_findings' or 'preview_fix' for previewing, which is part of the workflow.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67% with descriptions for severity_min and confirm. The description adds context for severity filtering ('optionally filtered by severity'), but the 'project_ref' param lacks explanation, and the description does not fully compensate for schema gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'bulk-apply' and resource 'all SQL fixes', clearly distinguishing from sibling 'apply_fix' for individual fixes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It advises to 'Always preview the count and list before confirming', providing a clear usage guideline. However, it does not explicitly state when to use this tool versus alternatives like 'apply_fix'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_findingsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the behavioral disclosure burden. It conveys that only findings from the 'last audit' are listed, implying a session context. However, it does not describe behavior in edge cases (e.g., no prior audit, multiple audits, result format, pagination, or idempotency). The basic intent is clear but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: first sentence defines the operation, second sentence provides usage guidance. No unnecessary words or repetition. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with 2 parameters and no output schema, the description covers the basic purpose and usage context. However, it omits details about the expected output (list of findings? fields?), error conditions (no audit found), and what constitutes 'last audit'. It is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds that 'severity' is an optional filter, which clarifies its purpose. However, 'project_ref' (the required parameter) is not explained; the description only mentions 'a project' without specifying what format or identifier is expected. Partial but insufficient compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List findings') and the resource ('from the last audit of a project'), with an optional filter on severity. This distinguishes it from sibling tools like 'audit_project' (which creates the audit) and 'apply_fix' (which applies fixes), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to use this tool after 'audit_project' to inspect specific issues. This provides clear sequencing and context, helping the agent decide when to invoke this tool over alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• apply_fixA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses that the tool applies changes (destructive), requires confirmation, and triggers an audit. More detail on side effects or permissions could improve, but current transparency is strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each adding essential information: purpose, requirement, and process. No fluff, front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema or annotations, the description covers the key context: prerequisite (preview_fix), requirement (confirm), and postcondition (audit). It lacks detail on return values or error states but is sufficient for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is low (33% with only confirm described). The description reinforces the confirm requirement but does not add meaning to project_ref or finding_index. While the description adds some value, it does not fully compensate for the undocumented parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'ACTUALLY APPLY a fix SQL to the project' with a strong verb and resource. It distinguishes itself from sibling tools like preview_fix and apply_all_fixes by implying it's for a single fix and requiring a prerequisite.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidelines: requires confirm=true, always run preview_fix first, and re-runs audit afterward. This provides clear when-to-use instructions and prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.