Server Quality Checklist

Tool Scores

• x80_probeC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds useful behavioral context beyond annotations: it executes a package, inspects syscalls, and returns 'a literal next call to make.' However, it does not disclose potential side effects of running an arbitrary package, failure modes, or what happens to the files_in inputs, which matters given the annotations are all false and don't carry a safety profile.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tightly written sentences with no filler. The main action is front-loaded, the evidence-based intent is clarified, and the unusual return value is stated directly. The prose earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with six parameters, no output schema, and uninformative annotations, the description is too thin. It doesn't explain the input file mechanism, timeout behavior, family selection, or the exact structure of the returned 'next call.' The concept is clear but operational detail is largely missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the six parameters. 'Package' vaguely maps to 'program', but args, stdin, family, files_in, and timeout_ms are completely unaddressed, leaving the agent without meaningful parameter guidance for a non-trivial schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action—run a package on the cheapest profile—and a distinct outcome: report the OS actually needed based on syscalls. This separates it from the sibling run tools conceptually, though it doesn't name them explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for OS-need detection from evidence, but gives no explicit when-to-use or when-not-to-use guidance. It does not reference any sibling tools or alternative approaches, so an agent must infer when this probe is preferable to x80_cpm_run, x80_dos_run, or x80_diff_run.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• x80_diff_runA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With all annotations set to false, the description carries the behavioral burden. It does disclose two meaningful traits: comparison is byte-for-byte, and normalization must be explicitly declared rather than automatic. However, it does not mention that guest and host programs are actually executed with potential side effects, how outputs are stored, or process-group kill behavior, leaving important behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact at three sentences, with the core operation in the first sentence and the motivating case in the second. The anecdote adds context without bloating the definition, though it is arguably not essential.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a complex tool with 6 top-level parameters, nested FileIn/GuestRun/reference objects, and no output schema, yet the description provides only a high-level summary. An agent would not know how to structure guest runs, supply file inputs via host_path vs content_b64, or interpret normalization and comparison options, so the description is incomplete for invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description maps the core roles—'same inputs' points to the inputs array, 'guest program' to guest, and 'host reference implementation' to reference—adding semantic value. But with schema description coverage at only 33%, it fails to clarify the remaining parameters: compare modes, normalization enum choices, reference.argv placeholders, and timeout semantics are left to the schema, which is sparse for them.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action—'Run the same inputs through a guest program and a host reference implementation and compare the outputs byte for byte'—which clearly identifies the tool's function and differentiates it from sibling run tools like x80_cpm_run and x80_dos_run. The added context about the CP/M .COM versus Python case reinforces the intended differential-testing use.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for differential testing—running the same inputs through guest and host—but never states explicitly when to prefer it over x80_cpm_run, x80_dos_run, or x80_probe. There are no exclusions or conditional routing statements, so an agent must infer the appropriate context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• x80_dos_runA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses useful behavioral details beyond the annotations: fresh-sandbox isolation, propagation of the DOS AH=4Ch AL value as the exit code, a measured rc=7 example, and the ambiguity of rc=1. However, it references exit_code_meaning, which is not present in the input schema, creating potential confusion.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the first sentence states the core behavior, and the second adds the most important caveat and sibling differentiation. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 12 parameters, nested objects, no output schema, and sparse annotations, this description is too thin. It does not explain arguments, environment, file inputs, assertions, collection behavior, timeouts, sandbox retention, or return values. It captures the central exit-code nuance but leaves too much for the agent to infer or discover elsewhere.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 25% with 12 parameters, so the description should compensate for undocumented parameters, but it does not. It introduces the vague term 'DOS package' and references a non-existent exit_code_meaning field, while core parameters like program and profile are already documented in the schema. It adds little semantic value for the many other parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action and resource: 'Run one DOS package to completion in a fresh sandbox.' It also explicitly contrasts itself with x80_cpm_run, making its role among siblings immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It directly differentiates from x80_cpm_run by noting that this tool has a meaningful exit code, and it explains the rc=1 ambiguity and how exit_code_meaning resolves it. This gives an agent a concrete selection criterion, though it does not discuss other sibling tools or state explicit when-not-to-use conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• x80_filesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already signal destructive behavior, and the schema documents overwrite semantics, so the description does not need to restate those. It adds two useful behavioral clarifiers—resolve runs nothing, and direction is intentionally in the op name to prevent editing the host copy—but it leaves backend side effects, via-mechanism tradeoffs, and error behavior to the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, action-first, with no filler. The first sentence enumerates the full operation set, and the second explains the core design convention in a way that directly prevents a known agent failure mode.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 17-parameter, 5-op tool, the description provides a strong high-level map, and the detailed schema fills in op-scoped parameters, via options, overwrite behavior, and machine/sandbox requirements. The main residual gap is that there is no output schema and the description doesn't describe return shapes, but this is a minor omission given the richness of the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 82%, the input schema already carries most parameter meaning, including op-specific applicability and mutually exclusive fields. The description itself does not add parameter-level detail beyond its high-level operation mapping, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names concrete operations with specific verbs and resources: moving files between host and guest, listing guest-visible files, showing open handles, and performing dry-run name resolution. It also highlights a critical design convention—direction lives in the op name, not a flag—which makes the tool's purpose and correct use unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear op-level guidance: to_guest/from_guest encode direction, resolve is the no-side-effect 'where would this come from?' path, and list/handles map to guest visibility and open files. It does not explicitly name sibling tools like x80_cpm_run or x80_dos_run as alternatives, so it stops short of a full when-not-to-use statement.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• x80_imagesA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (openWorldHint:true, idempotentHint:true), the description adds genuinely useful behavioral context: the tool verifies as well as downloads, the catalog is pinned, and — most valuably — downloads can never be triggered implicitly by another tool call. Nothing contradicts the annotations: readOnlyHint:false aligns with 'download', and openWorldHint:true aligns with the description positioning this as the server's sole open-world operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short sentences with zero filler. The action is front-loaded in the first sentence, the server-wide contrast in the second, and the single most important operational constraint ('Never called implicitly') closes it. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For selecting and invoking the tool, the description plus schema is nearly sufficient. The notable gap is that there is no output schema and the description never hints at what the agent should expect after 'verify' succeeds or fails — no return shape, error behavior, or verification outcome semantics. Given the tool's five parameters and non-trivial download behavior, a sentence on expected results would complete the picture.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 60% and the tool description itself says nothing directly about parameters; it only hints at them via 'pinned catalog' (tying to romwbw_version) and 'disk images' (tying to image_ids). The schema picks up the slack for image_ids (concrete catalog id examples), romwbw_version (pinning behavior), and the fifth boolean parameter requiring dry_run:false, but dry_run and timeout_ms remain undocumented anywhere. The description neither compensates for that gap nor repeats what the schema covers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource pair — 'Download and verify disk images from the pinned romwbw_disks catalog' — which precisely states what the tool does. It also differentiates the tool from its siblings (x80_files, x80_probe, x80_profiles, x80_cpm_run, x80_dos_run, x80_diff_run) by noting that 'every other tool in this server is openWorldHint:false', marking this as the only download-capable tool in the set.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description delivers a strong when-not rule: 'Never called implicitly — no tool call may trigger a download', which is critical operational guidance for an agent. It also frames the tool against the whole server ('Every other tool... is openWorldHint:false'). However, it stops short of naming specific sibling alternatives for common non-download scenarios (e.g., browsing files with x80_files or probing with x80_probe), so the full when-to-use mapping is left somewhat implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• x80_cpm_runA
  4.2/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  All annotations are false booleans, so the description carries the full disclosure burden — and it delivers richly. It reveals that no exit_code field exists by design, explains why across three backends including a measured 1-of-23 file extraction failure that still exited 0, and instructs that success must be asserted from stdout plus the file manifest. This prevents a real, otherwise-invisible failure mode.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, purpose front-loaded with the critical behavioral warning bolded immediately after. The long second sentence is dense with concrete evidence that justifies the warning, so every clause earns its place; nothing is filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 13-parameter tool with nested objects and no output schema, the description compensates for the missing output schema by defining the return envelope and covers the single most dangerous interpretation trap. Remaining gaps (eol_convert, keep_sandbox, cpu have no schema or description text) are largely self-explanatory from their names and enum values, so the definition is complete enough for confident invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 54%, and the covered parameters (profile, program, timeout_ms, default_mode, args, stdin) already have high-quality descriptions, notably default_mode's warning about cpmemu auto mode. The description adds little parameter-level detail but does frame what assert and collect produce ("your assertions", "manifest of the files it created"). Net neutral: no harm, no major added meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: "Run one CP/M package to completion in a fresh sandbox" and names the full return envelope (console output, file manifest, termination reason, assertions). The CP/M scope and intentional absence of exit_code clearly distinguish it from the DOS sibling x80_dos_run without needing to open the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied rather than stated: the CP/M scoping and the sibling list (x80_dos_run, x80_diff_run) make the intended domain reasonably clear, but the description never says when to pick this over an alternative or gives exclusions. The exit_code warning is interpretation guidance for after the call, not selection guidance for before it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• x80_profilesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With readOnlyHint=true and destructiveHint=false in annotations, the safety profile is already clear. The description adds meaningful behavioral context: it reports runnable profiles, accounts for backend differences, and warns about conditions that will trip up x80_open. This goes beyond simply restating the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences deliver the core purpose, a direct call-to-action, and a concrete failure scenario. The most important guidance ('Call this first') is front-loaded and bolded, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only listing tool with strong annotations and a parameter-rich schema, the description is complete: it names the returned fields, explains why the tool exists, and warns about the key operational risk. An agent can decide when to call it and what to expect without needing additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75%, so the schema already documents the parameters adequately. The description does not add much parameter-level meaning, but it also does not need to given the schema coverage; the listed output fields help explain what the family/profile/probe parameters influence.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') with a clearly bounded resource ('the machine profiles this installation can actually run') and previews the key fields returned. It is immediately distinguishable from sibling tools like x80_files, x80_images, x80_probe, and the run tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The bold 'Call this first' instruction gives explicit timing guidance, and the explanation about missing images or ROM/disk version mismatches causing x80_open failures clarifies why this preflight step matters. However, it does not state when to skip this tool or name alternative tools for related concerns.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.