Server Quality Checklist

Tool Scores

• execute_pythonA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  It adds critical context beyond the annotations: the code is actually executed, so malicious or unwanted code is a side effect; the container is no-network, read-only, and killed at timeout; the repaired code is what runs; and data is retained to improve the service. It also discloses auth and x402 payment behavior. No explicit contradiction with the annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long, but it is structured and front-loaded: it starts with what the tool does, then covers side effects, alternatives, auth, parameters, and returned values. The length is justified by the complexity of a side-effecting, execution tool with payment requirements and repair semantics, though some parameter details are summarized the schema already contains.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description, together with the output schema, provides everything needed to choose and call this tool correctly: behavioral constraints, alternatives, auth flow, argument semantics, edge cases like null fixed_code, and return shape. The agent is not left to guess or infer critical behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though schema description coverage is 100%, the description explains practical operational semantics: code is measured in encoded UTF-8 bytes, language is restricted to Python despite the schema enum, the deployment caps timeout_s at 30, and expected_output/examples are the mechanism for proving correctness rather than just execution. This guidance is not inferable from the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a concrete verb and resource: run the submitted code in a throwaway container and report exit code, stdout, and stderr. It also orients the agent by explaining that the tool does everything repair_python does and then executes the code, which clearly separates it from validate_python and repair_python.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It tells the agent exactly when to choose this tool: when proof of execution is needed, such as 'that it runs or that it does what it says'. It explicitly names the alternative tools, validate_python and repair_python, and notes neither runs anything, which prevents selection mistakes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repair_pythonA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds critical behavior beyond annotations: code is never run, fixes are only returned when provably safe, fixed_code is null when no fix is possible, and authentication/payment requirements are disclosed. It also states which options are ignored because nothing executes. Annotations already indicate read-only and non-destructive, and the description does not contradict them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but logically organized by sections (behavior, use cases, auth, arguments, returns). It front-loads the core distinction from siblings and then systematically covers constraints. It is somewhat verbose, especially around payment details, but every section contributes decision-relevant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema exists, and the description still explains the null fixed_code semantics, return fields, auth failure behavior, and constraint enforcement. Nothing an agent needs to call this tool correctly appears to be missing, including what happens with unsupported options.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description enriches every parameter: it details byte limits for code, explains that fragments shift line/column numbers, states language must be 'python' and other values elicit 400, and clarifies that options like timeout_s, examples, and expected_output have no effect in this mode. This goes far beyond raw schema fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: repair Python code, returning deterministic fixes in fixed_code and preserving the original when safety cannot be proven. It also explicitly contrasts itself with the sibling tools validate_python and execute_python, so an agent can distinguish it without opening the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit selection criteria: use when validation failed and you want a fix, not just a diagnosis. It also names the alternatives ('validate_python when the diagnosis is enough; execute_python when the fix has to be proven to run').

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_pythonA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Even though annotations already mark readOnlyHint=true and destructiveHint=false, the description goes far beyond them: it promises safety on untrusted code ('Safe on code you do not trust'), discloses that it never runs code, and highlights silent no-op options ('Ignored options are not refused, so a call that sets them looks like it worked'). It also reveals rate limits and auth behavior (HTTP 429, key requirement, x402 payment flow) and the fact that submitted code is retained to improve the service. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but every sentence earns its place. It is clearly structured with labeled sections (Auth, Arguments, Returns) and front-loaded with the core purpose before diving into details. No filler or redundant repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, the description does not need to enumerate return fields in depth; it still gives a high-level list. It covers input constraints, authentication requirements, call-limits, error behaviors, sibling tool relations, and retention. An agent has everything necessary to call the tool safely and correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though the schema already covers parameters 100%, the description adds critical meaning: it notes empty/large code fails with HTTP 400/413, line/column numbers count from the submitted fragment, the language must be exactly 'python' despite the broad schema enum, and that only transpile_to among options actual effects in this static tool. It explains that other options are ignored rather than rejected, which is not communicated by the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a precise verb and resource: 'Check Python source without running it' followed by an explicit checklist (parse, lint/ruff, type-check/mypy, AST security policy, credential scan). It explicitly distinguishes the tool from its siblings: 'repair_python to get the corrected source instead of the diagnosis; execute_python to prove the code runs.' No ambiguity remains about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It states exactly when to use this tool ('Use it on every Python file you generated or edited, before writing it to disk') and clearly names alternatives and their purpose. The when-not-to-use is implicit but clear: if you need a corrected file, use repair_python; if you need runtime proof, use execute_python. This is explicit enough to route an agent correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.