Server Quality Checklist

Tool Scores

• firepass_workerA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must fully disclose behavioral traits. It mentions the tool can read/write/edit files, run bash, and search with ripgrep/ast-grep/jq, implying destructive potential. It also notes file access is sandboxed to 'cwd'. However, it omits details on permissions, network access, cleanup, or recovery after failures, which are important for an autonomous coding agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a one-sentence opener stating the purpose, a sentence listing capabilities, and a bullet-style list for parameters. Every sentence adds value without redundancy. It is front-loaded and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (autonomous agent with many capabilities) and the presence of an output schema (unseen), the description is fairly complete. It covers what the tool does, its main capabilities, and parameter usage. However, it lacks guidance on when to use this worker versus sibling tools, and missing behavioral details like expected output format or error handling prevent a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must add meaning beyond parameter names. It does provide brief explanations: for 'prompt' it says 'The coding task,' for 'cwd' it says 'Working directory to sandbox file access to,' for 'context' it says 'Optional file contents, errors, or specs to pre-load,' and for 'max_iterations' it says 'Max tool-call rounds (default 60).' While this adds some value, the explanations are minimal and do not fully compensate for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Run a coding task with FirePass worker (Kimi K2.6 Turbo + tool loop).' It specifies the verb (run), resource (coding task), and the autonomous agent (Kimi K2.6 Turbo + tool loop). This directly differentiates it from sibling tools like 'firepass_researcher' (likely for research) and 'firepass_reviewer' (likely for review), as the worker focuses on autonomous coding tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use by listing capabilities like reading/writing files, running bash, and iterating autonomously. However, it lacks explicit guidance on when not to use it or alternatives among siblings (researcher, reviewer, trio). No 'when-to-use' or 'when-not-to-use' clauses, leaving room for ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• firepass_trioA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full burden. It mentions sandboxing and a fix loop but lacks details on side effects, permissions, or potential long runtime. Behavioral traits like file modifications or error handling are not disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: one purposeful sentence followed by a well-structured argument list. Every sentence adds value, no fluff, and the purpose is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (5 params, orchestration, output schema present), the description is adequate but not thorough. It omits details on the fix loop, output format, and potential runtime warnings. The output schema reduces burden, but gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description explains each parameter's meaning (prompt, cwd, context, max_iterations, max_review_rounds), adding value beyond the schema's titles. This compensates well for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs a full FirePass trio (research → implement → review → fix loop), using a specific verb and resource. It effectively distinguishes from sibling tools (firepass_researcher, firepass_worker, firepass_reviewer) by being the orchestrator.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: use this tool to run the complete trio workflow. It provides clear context but does not explicitly state when to use this vs. alternatives, such as using individual sub-agents directly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• firepass_researcherA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses key behaviors: autonomous iteration, read-only file access, specific search tools, and the underlying model. While it could mention timeouts or error handling, the disclosed traits are sufficient for an agent to understand the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loading the purpose and key constraints, followed by a clear parameter list. Every sentence adds value, with no unnecessary repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema (not shown), the description adequately covers input parameters and behavioral constraints. It could mention determinism or caching, but the provided information is sufficient for an agent to select and invoke the tool correctly. Complexity is moderate, and the description meets the need.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, but the description lists all 4 parameters with meaningful explanations (prompt as research question, cwd as sandbox directory, context as optional pre-loaded files, max_iterations as tool-call rounds with default). This compensates well for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs a research task using the FirePass researcher (Kimi K2.6 Turbo) with a read-only tool loop. It specifies capabilities (read files, search with ripgrep/ast-grep/jq/glob, iterate autonomously) and constraints (no file writes or shell commands), effectively distinguishing it from sibling tools like firepass_reviewer or firepass_worker.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for research/analysis tasks by stating its read-only nature and explicit exclusions ('No file writes or shell commands'). It does not explicitly list when to use vs alternatives, but the constraints provide clear guidance. Sibling tool names further hint at different purposes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• firepass_reviewerA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully discloses behavior: read-only, autonomous iteration, and structured review output. It mentions the default max_iterations (60) and that no writes or shell commands occur, which is transparent for a review tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a clear one-liner, a paragraph on capabilities, and a bullet list of args. Every sentence adds value, and the purpose is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters (2 required), 0% schema coverage, and an existing output schema, the description thoroughly covers all aspects: purpose, behavior, parameter semantics, and return format (blocking issues, suggestions, praise). It is complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds detailed explanations for each parameter: prompt (what to review), cwd (sandbox directory), context (optional pre-loaded content), and max_iterations (tool-call rounds). This adds significant meaning that the schema lacks.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs a code review with FirePass reviewer and specifies the model and read-only tool loop. It explicitly lists what it returns (blocking issues, suggestions, and what's done well), distinguishing it from a vague tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains what the tool can do (read files, search, iterate autonomously) and what it cannot do (no file writes or shell commands). This provides clear usage context, though it does not explicitly differentiate from sibling tools like firepass_researcher or firepass_worker.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.