Server Quality Checklist

Tool Scores

• deed_fmtA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses two key behaviors: the tool enforces a single canonical layout, and it returns parse diagnostics when parsing fails because there is no tree. This adds useful context about failure modes, though it doesn't mention side effects like whether it writes to a file.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one sentence with clear front-loading: it states the purpose first, then explains the failure behavior. It is economical and contains no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter formatter with no output schema and no annotations, the description is adequately complete. It explains both success and failure return scenarios, giving enough context to use the tool correctly. Minor gaps include not describing the formatting style or whether the output is idempotent, but these are not essential.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes the only parameter 'source' completely, so baseline is 3. The description adds context about how the source is used ('the file does not parse') but doesn't significantly extend the schema's meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool formats a Deed program into a specific layout, using the verb 'Format' and resource 'Deed program'. It also distinguishes itself from siblings like deed_check (checking) and deed_run (running) by focusing on formatting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied but not explicitly stated: the tool is for formatting Deed programs. No alternatives or when-not-to-use scenarios are mentioned, though the description's focus on formatting makes the use case clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deed_checkA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It thoroughly explains the output format (JSON lines), the types of lines (diagnostic and obligation), the meaning of tiers (proven, tested, guarded), and that silence indicates a well-formed program. This goes well beyond a basic description, though it stops short of discussing side effects or error cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose, then systematically explains the output format and semantics. While it is somewhat detailed, every sentence contributes necessary information for understanding the tool's behavior. It is structured logically and avoids irrelevant fluff, earning a solid score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description fully compensates by explaining the return format (JSON lines), the meaning of diagnostic versus obligation lines, the three tier values, and the significance of silence. For a single-parameter tool with no output schema, this description is complete and prepares the agent to interpret results correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter 'source' with a clear description ('The whole text of one Deed module.'), giving 100% schema coverage. The tool description does not add additional parameter-level detail beyond what the schema already provides, so it meets the baseline but does not exceed it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Check a Deed program and report what the compiler found.' This clearly states the tool's function and distinguishes it from siblings like deed_test or deed_run by focusing on compiler feedback. The output format description further reinforces the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: to check a Deed program for compiler findings and to understand whether obligations are proven, tested, or guarded. It does not explicitly mention alternatives or when not to use it, but the context is unmistakable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deed_explainA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses what the tool returns: 'the page that code carries: what it means, why the rule exists, and usually an example.' This goes beyond a simple tautology by specifying the structure of the output, including the caveat that an example is 'usually' present. It does not explicitly state it is read-only, but 'explain' strongly implies no mutation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action and example ('Explain one diagnostic code, such as DEED4025'), then briefly details the return content. Every word earns its place with no redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter explanatory tool with no output schema, the description is nearly complete. It states what the tool does, what it returns, and even a caveat ('usually an example'). It could be slightly more complete by noting error behavior for invalid codes, but given the low complexity, the description is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the parameter 'code', with the description 'A diagnostic code, like `DEED4025`.' The tool description adds only the qualifier 'one' (singular) and repeats the example format. Since the schema already fully explains the parameter, the description provides no additional semantic value, keeping the score at the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Explain one diagnostic code, such as DEED4025.' The verb 'explain' is specific, and the resource is 'diagnostic code.' It also distinguishes from sibling tools like deed_check, deed_test, and deed_run, which imply execution rather than explanation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool: when you need to understand a diagnostic code's meaning, rationale, and example. It does not explicitly name alternatives or state when not to use it, but the purpose is clear enough to differentiate from the action-oriented sibling tools. A clear context is provided without exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deed_reviewA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full transparency burden. It explicitly states that the operation is in-memory ('Both sides stay in memory'), has no file access ('opens no file'), holds no capability, and that the policy verdict is 'evidence, not a transport error.' This is strong behavioral disclosure, though it does not cover all possible failure modes or side-effect details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, front-loaded with the primary purpose, and every sentence adds meaningful detail: purpose, output, policy behavior, and memory/safety characteristics. There is no filler or redundant repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has moderate complexity (before/after arrays plus a nested policy object), no output schema, and no annotations, the description covers the key aspects: input comparisons, return receipt content, policy semantics, and absence of side effects. It does not detail the exact JSON receipt structure or explain array correspondence, but it is sufficient for an agent to invoke and interpret results at a high level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (before/after arrays and policy object fields are all described). The description adds contextual meaning around the policy object (its verdict is evidence, not an error), but mostly restates what the schema already provides. Baseline 3 is appropriate because the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Compare') and a resource ('the checked module set before a patch with the set after it'), making the core action clear. It also specifies the output ('one JSON review receipt') and distinguishes this from sibling tools like deed_check or deed_run by focusing on before/after patch comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly establishes the use case: comparing module sets before and after a patch. It does not explicitly mention when not to use the tool or name alternatives among the sibling tools, so it stops short of full guidance, but the context is unambiguous enough for an agent to decide on appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deed_fixA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool applies machine-applicable repairs, returns the repaired program, and defers uncertain suggestions to `deed_check`. This gives meaningful behavioral context, though it doesn't cover potential side effects or error handling, which keeps it just short of a 5.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, tightly packed with actionable information. The first sentence states the main action and result; the second clarifies the boundary with `deed_check`. No filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema, no annotations), the description covers all essential context: what it does, what it returns, and how it relates to the sibling tool for uncertain suggestions. Fully sufficient for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter `source` is fully described in the schema ('The whole text of one Deed module.'), with 100% schema coverage. The tool description adds no extra parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: 'Apply every machine-applicable repair the compiler offers' and the resource (the repaired program). It also distinguishes itself from the sibling tool `deed_check` by clarifying that uncertain suggestions appear there, not in `deed_fix`.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly defines when to use this tool: to apply every repair the compiler would make without asking. It also provides a when-not case: suggestions the compiler is not sure about are left and surface in `deed_check`, serving as a clear alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deed_runA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully carries the behavioral disclosure burden. It explicitly discloses that the tool refuses to run if the program 'does not check' or asks for capabilities the server does not hand out, and explains the filesystem limitation ('there is no filesystem here, so a program that reads or writes files is refused rather than failed'). This gives the agent accurate expectations about preflight failures and sandbox restrictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, with the core purpose in the first sentence and additional behavioral caveats in the second. Every sentence earns its place; there is no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema), the description is sufficient. It explains what the tool does, what it reports (printed output), and the preflight refusal conditions. The absence of an output schema means return values do not need to be detailed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already includes a clear description for `source` ('The whole text of one Deed module'), providing 100% coverage. The tool description adds no additional parameter semantics, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Run a Deed program's `main` and report what it printed.' It uses a specific verb (run) and resource (Deed program's main), distinguishing it from sibling tools like `deed_check` or `deed_test`, which likely validate or test rather than execute.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly establishes the tool's purpose as running `main`, which implies when to use it, but it does not explicitly mention how it compares to siblings like `deed_check` or `deed_test`. It does provide valuable usage boundaries by stating refusal conditions (e.g., if the program doesn't check or requests unsupported capabilities like file access), giving concrete context for when execution will be rejected.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deed_testA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully carries the transparency burden. It discloses the output format (JSON lines), the presence of a summary line, no-test behavior, that property lines are generated from `where` clauses and `ensures`, and that seeds are included for reproducibility. It even describes the refusal behavior. This is comprehensive and non-contradictory.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is composed of four sentences, each covering a distinct and necessary aspect: the core action, the output format, the property generation nuance, and the failure behavior. There is no redundancy or filler; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema, the description must explain the return format, and it does so thoroughly. It also covers edge cases (no tests), property seeds, and refusal behavior, making it self-contained and complete for an agent to correctly invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There is only one parameter, `source`, with 100% schema description coverage ('The whole text of one Deed module.'). The description does not add additional parameter-specific detail, but the schema already fully documents the parameter, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Run' with a clear resource 'Deed program's tests', and explicitly differentiates itself from sibling `deed_check` by directing users there for errors. It also clarifies that it covers both `test` blocks and generated properties, making the tool's scope unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use the tool (to run tests) and provides a clear when-not: 'Refuses without running when the program does not check: ask `deed_check` for what is wrong with it.' This direct alternative and exclusion is exactly what usage guidance needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.