Server Quality Checklist

Tool Scores

• forge_logsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and delivers comprehensive behavioral disclosure. It explicitly states 'READ-ONLY, idempotent,' describes file system behavior, security measures (path traversal guard), error handling (silent skip of corrupt lines), and operational characteristics (no authentication, no network, no rate limits).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, behavior, use cases, exclusions, returns, example) and every sentence adds value. It's comprehensive without being verbose, using bullet points and clear headings to organize information efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only query tool with 5 parameters and no output schema, the description provides complete context. It explains the tool's purpose, behavior, usage scenarios, exclusions, and detailed return format. The example demonstrates both input and output, compensating for the lack of output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description lists the filterable parameters in the opening sentence and provides an example showing parameter usage, but doesn't add significant semantic value beyond what the schema already documents. The description of 'runId' behavior when omitted is already covered in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Query the structured JSONL event stream that forge writes on every tool call throughout a run.' It specifies the exact resource (structured JSONL event stream) and action (query with filtering), and distinguishes itself from siblings by mentioning specific alternatives like 'session_state' and 'memory_save'/'memory_recall'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' scenarios with four specific use cases (debugging, auditing, orchestration confirmation, investigation) and 'Do NOT use for' guidance with three clear exclusions (live progress display, appending entries, long-term knowledge). It names alternative tools for each exclusion case.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• iteration_stateA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and delivers comprehensive behavioral disclosure. It details the three action types (READ, MUTATION), file storage locations, runId validation with regex pattern, security guarantees against path traversal, authentication/network/rate limit status, and get behavior for unknown modules. It also explains version differences in state scoping and distinguishes safe vs side-effect operations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, behavior, usage guidelines, returns, example) and every sentence adds value. While comprehensive, it's appropriately sized for a complex tool with multiple operations. The front-loaded purpose statement immediately communicates core functionality, though some behavioral details could be more condensed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with 4 parameters, no annotations, and no output schema, the description provides exceptional completeness. It covers purpose, behavior, usage scenarios, exclusions, security aspects, version differences, return formats for all three actions, and includes a concrete example. The description fully compensates for the lack of structured metadata, making the tool's functionality and constraints completely understandable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3. The description adds meaningful context beyond the schema by explaining the runId's purpose in scoping state to forge runs and legacy behavior implications, clarifying that update action bumps counters and flags, and providing concrete examples of moduleId values. However, it doesn't elaborate on specific parameter interactions or edge cases beyond what the schema already documents.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool's purpose as 'Read, update, or reset the per-module retry state for a forge run' with specific details about what it tracks (attempt count, score history, last status, last root cause, stagnation flag). It clearly distinguishes this from sibling tools by contrasting with session_state for cross-module reasoning and validate for recording validation attempts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' scenarios with three concrete examples (orchestrator retry decisions, debugger inspection, resetting for fresh runs) and 'Do NOT use for' guidance that names specific alternative tools (session_state for cross-module reasoning, validate for recording attempts). This gives clear context for when to choose this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• memory_saveA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and delivers rich behavioral details: it discloses this is a MUTATION tool (file append), describes deduplication logic, idempotency, atomic writes for concurrency, and operational characteristics (no auth/network/rate limits). It also explains the return format and duplicate handling behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Behaviour, Use when, Do NOT use, Returns, Example) and front-loads the core purpose. While comprehensive, some sentences could be more concise (e.g., the duplicate explanation is slightly verbose). Overall, most content earns its place by adding value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description provides complete context: it explains the tool's purpose, behavioral characteristics, usage guidelines, parameter context, return values, and includes a concrete example. This gives the agent everything needed to correctly select and invoke this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds meaningful context beyond the schema: it explains that patterns are stored as JSONL entries with timestamp (not in schema), clarifies duplicate detection is case-insensitive, and provides concrete examples of pattern usage (test commands, conventions, failure patterns) that help understand parameter semantics in practice.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('persist', 'store') and resources ('learned pattern', 'JSONL entries'), distinguishing it from siblings like session_state (ephemeral) and forge_logs (event logs). It explicitly defines what constitutes a pattern and how it's stored.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' scenarios (Phase 5 Learn, debugger root cause, reviewer conventions) and 'Do NOT use for' exclusions (ephemeral state, retry history, logs, large blobs), with clear alternatives named (session_state, iteration_state, forge_logs). This gives comprehensive guidance on when to choose this tool over siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validateA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and delivers comprehensive behavioral disclosure. It details mutation behavior (appends to iteration state, emits events), execution constraints (timeouts, shell execution), authentication/network/rate limit status, cwd precedence rules, and failure handling for nonexistent paths.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Behavior, Use when, Do NOT use, Returns, Example) and front-loaded core purpose. While comprehensive, some sentences could be more concise (e.g., the cwd precedence explanation is verbose). Overall, most content earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex mutation tool with 6 parameters, no annotations, and no output schema, the description provides exceptional completeness. It covers purpose, usage guidelines, behavioral details, parameter context, return structure with example, and distinguishes from all relevant sibling tools, leaving no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds meaningful context beyond schema: it explains cwd precedence rules, consequences of nonexistent cwd, that commands use shell execution allowing pipes/redirects, and that contract checks verify importer references match exporter exports. However, it doesn't fully explain all parameter interactions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the verb ('Run full verification') and resource ('forge module'), clearly distinguishing it from siblings like 'validate_plan' for plan-level checks. It specifies the comprehensive scope including verification commands, file existence, syntax validation, and API contract checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes explicit 'Use when' scenarios (e.g., after DONE report, after retry, manual validation) and 'Do NOT use for' exclusions (e.g., plan-level checks, querying past attempts), naming specific alternative tools like 'validate_plan', 'forge_logs', and 'iteration_state'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_planA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure and excels. It details the tool's read-only nature, event emission, no authentication/network/rate limits, error-handling approach (never throws, returns arrays), and optional parameter behavior. This covers safety, side effects, and operational constraints thoroughly beyond what a schema could convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Behavior, Use when, Do NOT use, Returns, Example) and front-loaded key information. While detailed, every sentence adds value (e.g., explaining validation checks, usage contexts, return structure). Minor verbosity in listing all checks keeps it from a perfect 5, but it remains efficient and organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (validation with multiple checks) and lack of annotations/output schema, the description is highly complete. It explains the validation scope, behavioral traits, usage guidelines, parameter semantics, and return structure with examples. No critical gaps exist; an agent has all needed context to invoke and interpret results correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the baseline is 3. The description adds significant value by explaining the optional parameter's behavior: 'planPath is optional; when omitted, the most recently modified file in .forge/plans/ is used.' This clarifies the default logic, enhancing understanding beyond the schema's basic description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('structurally validate a forge plan JSON file') and resources ('plan JSON file'), distinguishing it from siblings like 'validate' (for single module build output) and 'iteration_state' (for attempt counts). It explicitly lists the validation checks performed, making the purpose highly specific and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' scenarios (e.g., after planner writes a plan, debugging execution order issues, human pre-flight checks) and 'Do NOT use for' exclusions (e.g., executing a plan, validating single module output, inspecting retry history), with named alternatives like 'validate' and 'iteration_state'. This gives comprehensive guidance on when to use this tool versus others.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• memory_recallA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure and excels at it. It explicitly states: 'READ-ONLY, idempotent. No telemetry side effects, no access counters bumped, no state mutated. No authentication, no network, no rate limits.' It also describes file sources, empty result behavior ('never throws'), and implementation details (JSONL store, keyword matching). This provides rich behavioral context beyond basic functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, behavior, usage guidelines, exclusions, returns, example) and every sentence adds value. It's appropriately sized for a tool with rich behavioral context and sibling differentiation, with no redundant or wasted text. The information is front-loaded with the core purpose first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, the description provides exceptional completeness. It covers purpose, behavioral traits, usage scenarios, exclusions, return format (text block with grouping and sorting), and includes a concrete example. The only minor gap is not explicitly documenting the exact output schema, but the return description and example adequately compensate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so the baseline is 3. The description adds meaningful context: it explains that 'query' searches against 'pattern text, category name, and any included tags' (not just keywords), and clarifies that 'scope' determines which files are read (project.jsonl vs. global.jsonl). However, it doesn't provide additional syntax or format details beyond what the schema already documents.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search forge's learned-pattern memory for entries relevant to a query.' It specifies the exact resource (learned-pattern memory) and verb (search), and distinguishes it from siblings like memory_save (for saving), iteration_state (for retry history), and forge_logs (for structured events).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' scenarios with three concrete examples (planner agent decomposition, debugger analysis, worker agent decision-making) and a 'Do NOT use for' section that names four specific alternatives (memory_save, iteration_state, forge_logs, and full-text search). This gives comprehensive guidance on when to use this tool versus others.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• session_stateA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and delivers exceptional behavioral transparency. It details mutation/read behavior per action, file storage location, atomic write semantics, runId validation pattern, timestamp stamping, sorting behavior, and explicitly states 'No authentication, no network, no rate limits.' This provides complete behavioral understanding beyond what parameters alone would convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured with clear sections (Behaviour, Use when, Do NOT use for, Returns, Example). Every sentence earns its place by providing essential information without redundancy. The front-loaded purpose statement immediately communicates the tool's value, followed by organized details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (3 parameters, no output schema, no annotations), the description provides complete context. It covers purpose, behavior, usage guidelines, parameter semantics, return values, and includes a detailed example. The description fully compensates for the lack of output schema and annotations, making the tool completely understandable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3, but the description adds significant value. It explains the expected shape of the state parameter with detailed field descriptions, clarifies when runId is required/ignored, and provides a comprehensive example showing parameter usage. This goes well beyond what the schema provides alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs (save, load, list) and resource (orchestrator session snapshots). It distinguishes from siblings by explicitly mentioning what NOT to use it for (iteration_state, memory_save), showing clear differentiation from other tools on the server.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' scenarios with three concrete examples and 'Do NOT use for' guidance with three specific alternatives. It clearly defines when to use this tool versus sibling tools like iteration_state and memory_save, offering comprehensive usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.