Server Quality Checklist

Tool Scores

• save_chainA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden and does well: it discloses pre-write validation, atomic failure ('nothing is written on validation failure'), the name-to-frontmatter equality requirement, and the overwrite-gated replacement. These are meaningful behavioral details beyond a generic 'save' operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the core action, followed by validation and overwrite rules. Each sentence contributes operational information, though the overwrite reference could have been stated more explicitly as a parameter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool, the description covers validation and replacement behavior but leaves the exact content structure underspecified and omits the overwrite parameter from the schema, making the contract incomplete. An agent might struggle to construct a valid call without additional inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds value by explaining that content is a Markdown file with YAML frontmatter and listing the frontmatter fields, which the schema's 'content' property does not document. However, it references an 'overwrite: true' option that does not appear in the input schema, leaving ambiguity about where that flag lives.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Create or update a custom thinking chain as a Markdown file with YAML frontmatter.' This clearly distinguishes it from sibling operations like run_chain, advance_chain, export_chain, and get_chains by framing it as the write/persist operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The create/update intent is unambiguous and the overwrite condition gives a usage constraint, but the description never explicitly contrasts this tool with siblings or states when a user should prefer save_chain over run_chain or export_chain. Usage context is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• advance_chainA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of explaining behavior. It does disclose useful traits: idempotent retries via expected_stage_index and the mutual exclusivity of stage_output and skip_reason. However, it does not describe side effects, state changes, permissions, or error behavior, so the behavioral picture remains incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences, front-loaded with the main purpose before mode-specific details. Every clause carries useful information: purpose, staged usage, freeform usage, and retry behavior. No filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Normal invocation is well-covered: required IDs, the output/skip choice, freeform completion, and retries. But since there is no output schema and no annotations, the return value shape, failure modes, and stateful edge cases are left underspecified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds semantic meaning beyond the schema: it maps stage_output and skip_reason to chain modes, explains done in the freeform case, and frames expected_stage_index as a retry guard. Schema coverage is already 80%, but the description enriches parameter understanding rather than just restating names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence names a specific action ('Submit the current stage's result') and outcome ('receive the next stage'), making the tool's core purpose immediately clear. The staged/freeform distinction further scopes the behavior and separates this tool from listing, running, saving, or exporting chains.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context: this tool is called to submit current-stage work in a chain and get the next stage. It also explains when to use stage_output vs skip_reason and how the freeform chain differs, but it does not explicitly route away from sibling tools such as run_chain or save_chain.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_chainsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden of behavioral disclosure. It does so by detailing sources, precedence, and additional diagnostics (invalid chain files, missing packs, pack conflicts, shadowing). It doesn't explicitly state non-mutation, but 'List' strongly implies read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every clause carries distinct information: sources, paths, precedence, and diagnostics. It is front-loaded with the main verb and object, then adds structured detail without filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter enumeration tool with no output schema, the description covers what is needed: what is listed, where chains come from, layering precedence, and what anomalies are reported. No critical selection or invocation information is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema fully describes this with 100% coverage, so there are no parameter semantics to clarify. The baseline of 4 applies because the description correctly adds no unnecessary parameter noise.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('List') and a concrete resource ('all available thinking chains'), then expands with built-ins, file-defined layers, and shadowing. The operation is clearly distinct from siblings (run, advance, save, export), which are actions on a chain rather than enumeration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description establishes the tool as the inventory/read operation for chains, including diagnostics and shadowing details. It doesn't explicitly name alternatives or when-not-to-use, but the sibling names make the boundary inferable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_chainA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral disclosure burden. It does disclose important behaviors: rerouting through master, single-shot checklist returns, session-based freeform chains, and continuation via advance_chain. It stops short of detailing the exact session-start return value or error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences long, and every sentence earns its place: start action, routing guidance, checklist behavior, and session continuation. The core purpose is front-loaded and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with no output schema, the description covers start, routing, and next-step behavior well. The main omissions are a fuller explanation of what PASS means and what the immediate return value is when a session is opened.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers the only parameter with examples, and the description reinforces and extends that meaning by explaining how "master" and "default" are used and tying custom chains to list_chains. This adds practical routing context beyond the schema's basic name field.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'Start a thinking chain by name.' It differentiates from siblings by explicitly naming advance_chain for continuation and list_chains for discovering custom chains, so an agent can tell which tool to use.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit routing guidance: call a matching chain directly when the task shape matches, otherwise use run_chain("master") to route between PASS, "default", and custom chains. It also clarifies that checklist chains are single-shot while staged/freeform chains open a session and should be continued with advance_chain.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_chainA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the burden. It discloses what the tool returns (content, sha256, suggested filename), which sets expectations for a read/export operation. It does not explicitly state that nothing is mutated, but 'export' and the return payload imply a snapshot operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: one for the export purpose and return values, one for the import workflow, one for the scope edge case. No filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only export with two simple parameters, the description covers the return format, the import counterpart, and the only tricky parameter (scope ambiguity). With no annotations and no output schema, this is sufficient for an agent to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain the parameters. It does explain the scope parameter's disambiguating purpose and makes 'name' obvious as the chain identifier from context. It does not describe the name format, but that is low-risk.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb (export) and resource (a chain's raw Markdown source) and the purpose (for sharing). The mention of return values (content, sha256, suggested filename) distinguishes it from siblings like run_chain or advance_chain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to call save_chain with the content when importing a shared chain, which is an clear alternative. It also gives a concrete condition for when to pass scope: 'If the name exists in both global and project scope.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.