Server Quality Checklist

Tool Scores

• compare_energyA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool measures energy and reports margin, implying a read-only operation, but does not explicitly disclose non-destructiveness or any behavioral traits like permissions. Adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise, front-loaded sentences with no waste. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters, no output schema, and no annotations, the description is insufficient. It omits return format, parameter details, and how the measurement works, leaving the agent underinformed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must explain parameters. It fails to describe 'repeats', 'idle_seconds', or the meaning of command_a and command_b beyond their names. No parameter semantics are provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool measures two commands to compare energy consumption and reports the margin. The verb 'measure' and resource 'energy' are specific, and the purpose is distinct from siblings like verify_equivalence and compare_tokens.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the tool is only meaningful if commands are functionally equivalent and advises to verify equivalence separately, pointing to the sibling tool verify_equivalence. This provides clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_tokensA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full behavioral transparency burden. It clearly describes what the tool does (compares token usage) but does not detail how tokens are counted or if there are any side effects. The behavior is straightforward and non-destructive, so transparency is high.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two sentences, each essential. The first states the core function; the second adds critical context. No superfluous content. Perfectly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (4 params, no output schema). The description explains the main output (which program uses fewer tokens and by how much) but lacks specifics on the exact format (e.g., token counts, percentage). Close to complete but leaves a minor gap for output details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description provides no explanation of the parameters (command_a, command_b, upstream, base_url_env). It fails to clarify how programs are specified or the role of optional parameters, leaving the agent without necessary semantic guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's verb (measure/compare) and resource (token use of two programs). It distinguishes it from siblings like 'compare_energy' and 'verify_equivalence' by focusing on token comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use the tool (to compare token usage) and provides a critical precondition: programs must produce equivalent results. It warns against using it when results are not equivalent and hints at using 'verify_equivalence' separately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• measure_energyA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description fully discloses the methodology: sampling CPU power at 100ms, subtracting idle baseline, repeating runs (default 3x), and reporting mean/stdev/CV. It also notes expected wall time, providing clear behavioral insight.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a clear first sentence, method details, usage notes, and a time estimate. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and lack of output schema, the description covers measurement methodology, parameters, output statistics, and wall time. It is largely complete, though the return format of the statistics is not explicitly described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates well by explaining the purpose of 'repeats' (default 3x) and 'idle_seconds' (idle baseline). It also advises on handling paths, adding meaning beyond the schema. However, it does not detail all parameters' formats or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: measuring electrical energy consumed by a shell command. It uses a specific verb ('measure') and resource ('electrical energy in joules'), and distinguishes from siblings like 'compare_energy' by focusing on a single command.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for measuring energy of a shell command but does not explicitly state when to use this tool versus alternatives like 'compare_energy' or 'energy_backend_info'. No exclusions or when-not-to-use guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• measure_tokensA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully reveals internal behavior: it sets up a non-blocking counting proxy, runs the program, and reports usage. It also explains the consequence of zero llm_calls and the env var dependency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single paragraph of about 100 words, efficiently covering purpose, mechanism, usage tips, and edge cases without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains what will be reported (input/output/total tokens, call count) and addresses the zero-call scenario. It covers all necessary aspects for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds meaning to 'base_url_env' and 'upstream' by explaining their roles in the proxy mechanism. 'command' is implied but not detailed. Partially compensates for low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb 'measure' and the resource 'LLM tokens consumed by a program'. It differentiates from sibling tools like measure_energy by focusing on tokens and mentions the proxy-based method.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides context on when to use: for measuring token consumption of programs that read an LLM endpoint from an environment variable. Gives guidance for different providers and explains the zero-call edge case. However, does not explicitly state when not to use or list alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_equivalenceA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries full burden. It explains the comparison logic and input modes. However, it does not disclose that the commands may have arbitrary side effects (destructiveness), which is a notable gap for a tool that executes arbitrary commands.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four concise, well-structured sentences. Purpose is front-loaded, usage variations are logically separated, and no unnecessary text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, usage, and parameter semantics well. However, it does not describe the return value (e.g., boolean, diff output) despite having no output schema. This is a minor but relevant omission.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It effectively explains the stdin_inputs parameter and its behavior (list of strings fed to stdin, requiring all outputs to match). The command_a and command_b parameters are only implied, but the tool's purpose is clear enough.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs two commands and compares stdout and exit code. It also distinguishes itself as a gate before energy/token comparisons relative to siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly positions itself as a preliminary gate before energy/token comparisons. Describes two usage modes (single-input smoke test vs. input battery) and recommends the project's own test suite for stronger evidence.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• energy_backend_infoA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full behavioral disclosure. It explains the condition for returning energy_available: false and the recommended alternative action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description covers the main return value (energy_available) and usage context, but could slightly benefit from more precise return format details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist and schema coverage is 100%. With 0 parameters, the baseline is 4, and no additional semantics are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it reports energy measurement availability and backend, and distinguishes from sibling tools like measure_energy by advising to call this first.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Call this before measure_energy' and provides clear guidance on what to do if energy is not available (rely on token axis or state can't measure).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.