Server Quality Checklist

Tool Scores

• cocotb_generate_runnerB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It mentions the output is 'deterministic' and 'standard', but does not disclose whether the tool writes a file to disk, returns Makefile content, overwrites existing files, or what side effects occur. This is a meaningful transparency gap for a generator tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the core purpose and includes no filler or redundant phrasing. Every word contributes to the meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema and no annotations, so the description should clarify what the caller receives. It is ambiguous whether 'generates' means writing a Makefile to disk or returning its contents, and there is no mention of file paths, overwrite behavior, or the deterministic output's meaning. The description is too thin for a tool an agent must invoke correctly without further context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already documents all four parameters. The description adds the context that the tool targets a Verilog DUT and Python test module, which maps to the parameters, but it does not add deeper semantic detail beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Generates'), a specific resource ('Cocotb simulation Makefile'), and key inputs ('Verilog DUT and Python test module'). This clearly differentiates the tool from its siblings like cocotb_run, which would execute simulations rather than create build artifacts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives such as cocotb_run or cocotb_list_tests. The description implies the tool is for producing a Makefile, but it does not state prerequisites, when generation is appropriate, or when another sibling should be chosen.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cocotb_runA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It transparently states the main flow — compile, execute, parse — and even names the output shape. However, it omits side effects for a tool that compiles and runs external simulators: build artifacts, VCD files when dump_waves is set, spawned processes, timeout termination behavior, and compile-failure handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single well-structured sentence with zero filler. Each clause earns its place: compilation, execution, then result parsing with a specific output detail. The main action is front-loaded and the sentence is easily scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex 7-parameter tool with no annotations and no output schema, the description covers the core flow and hints at the return value (JSON reports with tracebacks), which is a good start. But it is incomplete: no mention of prerequisites (simulator binaries, cocotb installation), filesystem side effects, failure modes, or how tests are discovered within python_module.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so every parameter already has a description, yielding a baseline of 3. The tool description adds marginal pipeline context (e.g., 'compiles the Verilog DUT' maps to verilog_sources and toplevel) but does not clarify parameter formats, defaults, or constraints beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names three specific actions (compiles, executes, parses), specific resources (Verilog DUT, cocotb testbench), and a concrete output (structured JSON reports with assertion tracebacks). This clearly distinguishes it from siblings like cocotb_list_tests, cocotb_generate_runner, and cocotb_toolchain_info, which cover discovery, scaffolding, and environment info rather than the full simulation run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The pipeline description implies usage: call this when you want to compile a DUT, run a testbench, and get JSON results. However, it never states when to prefer this over siblings (e.g., list tests first with cocotb_list_tests, generate a runner with cocotb_generate_runner) and gives no exclusions or prerequisites such as simulator installation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cocotb_list_testsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral disclosure burden. It does disclose the main guarantee ('without executing simulation') and the scope of results, which is helpful. However, it omits the return format, whether the file is parsed versus imported (potential module-level side effects), and error behavior, leaving meaningful gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence states the operation, target, and key non-execution guarantee with no filler. Every word earns its place, and the structure is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter tool, the description plus 100% schema coverage is nearly sufficient: it states what the tool returns conceptually (test coroutines and docstrings) and that it does not run simulation. It is missing an explicit routing note to cocotb_run and a precise output format, but the remaining gaps are minor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so both test_file and cwd are already documented in the input schema. The description only reinforces that a Python testbench file is the target and adds no new parameter semantics, matching the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Discovers') and identifies the exact resource: all @cocotb.test() coroutines and their docstrings in a Python testbench file. The phrase 'without executing simulation' clearly separates it from siblings like cocotb_run, so an agent knows exactly what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: use it when you need to inspect test cases and docstrings without running simulation. It does not explicitly name the alternative tool or provide when-not-to-use guidance, so it earns a 4 rather than a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cocotb_toolchain_infoA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The verb 'Returns' implies a read-only query with no side effects, which is reasonably transparent for a zero-parameter info tool. However, it does not explicitly state whether any environment setup or simulator connection is required, or how failures are reported.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence that states the exact return value categories with no filler or repetition. Every word earns its place, and the structure is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has zero parameters and no output schema, and the complexity is low. The description fully covers what the tool returns (runtime environment plus three version types), which is sufficient for an agent to call it correctly and interpret the result at a high level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters and schema coverage is trivially 100%, so no parameter documentation is needed. The description adds no parameter meaning, but none is required; the baseline of 4 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Returns') with a clear resource: active container/host runtime info and versions of Cocotb, Python, and simulator. It immediately identifies this as an info/query tool, which distinguishes it from the sibling tools that run tests, list tests, or generate runners.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies the tool is for retrieving environment/version details, but it does not explicitly state when to choose it over alternatives or mention any exclusions. An agent can infer the use case from the return value description, but there is no explicit routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.