Server Quality Checklist

Tool Scores

• tokenizeB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool converts code to tokens but doesn't disclose behavioral traits such as error handling (e.g., invalid code), performance characteristics (e.g., speed, limits), or side effects. The description is minimal and lacks critical operational details beyond the basic function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose clearly. The 'Args' and 'Returns' sections are structured efficiently with no redundant information. Every sentence earns its place, making it concise and well-organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (tokenization task), no annotations, and an output schema present (which handles return values), the description is partially complete. It covers the basic purpose and parameters but lacks usage guidelines, behavioral details, and context for integration with siblings. It's adequate as a minimum viable description but has clear gaps in guidance and transparency.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds some meaning beyond the input schema: it specifies that the 'code' parameter must be 'Valid Pylpex source code', which clarifies the expected input format. However, with 0% schema description coverage and only one parameter, the description compensates partially but doesn't provide detailed semantics (e.g., code examples, tokenization rules). Baseline is 3 due to low parameter count and minimal added value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Convert Pylpex code into tokens for analysis.' It specifies the verb ('Convert'), resource ('Pylpex code'), and outcome ('tokens for analysis'), which distinguishes it from siblings like 'get_variables', 'reset', and 'run'. However, it doesn't explicitly differentiate from siblings beyond the basic function, missing a direct comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, context for tokenization (e.g., preprocessing steps), or comparisons to sibling tools. Usage is implied by the purpose but lacks explicit when/when-not instructions or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_variablesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It states the tool retrieves variables but doesn't specify format (e.g., dictionary, list), scope (e.g., local/global), or any side effects (e.g., performance impact). This leaves significant gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without any wasted words. It is front-loaded and appropriately sized for its simple purpose, earning full marks for conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, 100% schema coverage, and an output schema exists, the description is minimally adequate. However, with no annotations and sibling tools present, it lacks context on usage and behavioral details, making it incomplete for optimal agent guidance despite the structured data support.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately doesn't discuss parameters, and the baseline for this scenario is 4, as it avoids redundancy while being complete for a parameterless tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('all variables in the current interpreter state'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'reset' or 'run', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'reset' (which might clear variables) or 'run' (which might execute code affecting variables). It lacks explicit when/when-not instructions or context for selection among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resetB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states what the tool does ('reset the interpreter state') without explaining what 'reset' entails (e.g., clears variables, resets execution environment, irreversible changes). It lacks details on permissions, side effects, or response behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words, making it appropriately sized and front-loaded for a simple tool. Every word contributes to the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, 100% schema coverage, and an output schema exists, the description is minimally adequate. However, as a mutation tool with no annotations, it should better explain what 'reset' means behaviorally (e.g., what gets cleared, if it's reversible) to be more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100%, so no parameter documentation is needed. The description doesn't add parameter details, which is appropriate, earning a baseline score of 4 for this dimension.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('reset') and the target ('interpreter state'), which is a specific verb+resource combination. However, it doesn't differentiate from sibling tools like 'get_variables' or 'run', which might also affect interpreter state in different ways.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'run' (which might execute code) or 'get_variables' (which might inspect state). The description implies it's for resetting state but doesn't specify contexts or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly describes that this tool executes code and returns results, mentions the need to follow syntax conventions, and provides example outputs. It doesn't cover potential side effects, security implications, or error handling, but provides substantial behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose. Every sentence adds value: the main execution statement, three prerequisite steps, parameter documentation, return value explanation, and illustrative examples. No wasted words or redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a code execution tool with no annotations, the description provides comprehensive context. It explains the tool's purpose, prerequisites, parameter semantics, return values, and includes examples. The presence of an output schema means the description doesn't need to detail return format, allowing it to focus on execution behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description must compensate fully. It explicitly documents the single parameter 'code' as 'Valid Pylpex source code to execute', provides syntax guidance, and includes concrete examples showing how the parameter should be formatted and what constitutes valid input.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb 'Execute' and resource 'Pylpex code', distinguishing it from siblings like get_variables (retrieval), reset (state management), and tokenize (analysis). It provides a complete purpose statement with the expected outcome.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool (for executing Pylpex code) and includes prerequisites (check documentation, review examples, ensure syntax conventions). However, it doesn't explicitly state when NOT to use it or name specific alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.