Server Quality Checklist

Tool Scores

• python_codeC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses execution and return output but does not mention side effects, environment constraints, security implications, or error behavior common for code execution tools.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single concise sentence with the key action front-loaded. Slight filler 'This function will...' prevents a 5, but it is otherwise efficient and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a code execution tool with no annotations and no output schema, the description is too thin. It lacks critical context about side effects, environment, and how it differs from similar tools like python_file or bash.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description must compensate, but it only says 'python code' which is already implied by the tool name. No details on code formatting, execution context, or output handling are given.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it executes Python code and returns output, using specific verb 'execute' and resource 'python code'. However, it does not explicitly distinguish itself from sibling tools like python_file or bash.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool over alternatives such as python_file or bash. It only describes what the tool does without any context for selection or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• bashC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosure. It only mentions execution and output, but fails to note that bash commands can modify the filesystem, have side effects, or require privileges. There is no mention of error handling, exit codes, or potential destructive behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short but contains redundancy. The second sentence repeats 'executing bash commands' and lists examples that add little value. It earns a middle score: acceptable length but not optimally concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that can execute arbitrary commands, the description is severely incomplete. It does not cover return format, error behavior, environment, or security considerations, and lacks any comparison with sibling tools. The presence of an output schema or annotations would mitigate this, but neither exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% coverage for the 'command' parameter, and the description adds minimal meaning beyond the tool name. It implies the parameter is a bash command string but provides no syntax, examples, or constraints, leaving the agent without sufficient semantic detail for reliable invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool executes a bash command and returns the output. The verb 'execute' and resource 'bash command' are specific, and it is distinct from siblings like read_file, write_file, python_code, etc., which handle other operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like python_code or the file-specific tools. The statement 'useful for executing bash commands including creating files, directories' is tautological and does not offer exclusions or conditions for choosing this tool over siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• grepC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It only states the basic operation (search and return matching lines) but does not disclose whether the pattern is a regular expression, if matching is case-sensitive, what happens if the file is missing, or whether line numbers are included. These are significant behavioral unknowns.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, readable sentence that front-loads the core verb and resource. It is concise and to the point, though it could be slightly tighter without 'This function will'.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, no output schema, no annotations), the description is minimal but insufficient. It does not explain the return format, error handling, or pattern semantics, which are essential for an agent to invoke the tool correctly. The description feels like a skeleton rather than complete guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain parameters. It merely labels 'pattern' and 'file' without adding details (e.g., file path format, regex syntax, escape characters). The description adds little beyond the parameter names themselves.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('search for the given pattern'), the resource ('specified file'), and the outcome ('return the matching lines'). It distinguishes the tool from siblings like read_file (which reads whole file) and write_file (which writes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention exclusions, prerequisites, or contexts where grep is preferred over read_file or bash. The only usage hint is implicit in the action itself.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• python_fileC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosure. It only states that it executes a file and returns output, omitting critical information about potential side effects, sandboxing, working directory, error handling, or security implications of running arbitrary code.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one sentence and front-loaded, making it easy to scan. However, the phrase 'This function will' is filler; a more direct phrasing like 'Executes a Python file and returns output' would be tighter. The conciseness is good but not maximal.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that executes arbitrary code, the description is severely under-specified. It lacks details about the execution environment, return format, error behavior, and access to system resources. Given the high-risk nature and absence of annotations or output schema, the description is insufficient for safe and correct use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one required string parameter 'file' with no description. The description adds that it is a Python file, implying it is a path, but does not clarify whether it accepts absolute/relative paths or file content. It provides minimal additional meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action (execute) and resource (a python file) and mentions returning output. However, it does not differentiate from the sibling tool python_code, which also executes Python, but the purpose itself is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool over alternatives like bash or python_code. The description does not mention use cases, prerequisites, or exclusions, leaving the agent without contextual decision-making information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_fileB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It only states the basic function and return type, but omits important behaviors like error handling for missing files, binary file handling, or any permission requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It efficiently communicates the tool's purpose and return type.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool, the description provides the core purpose and return type, but lacks details about error scenarios, encoding, or size limits. With no annotations or output schema, more behavioral context would help, though the low complexity keeps it from being severely inadequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description implicitly clarifies that the 'file' parameter refers to the file to be read, but adds no detail about path format, file type, or constraints. With 0% schema description coverage, the description offers minimal compensation beyond the parameter name itself.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads a file and returns its contents as a string, with a specific verb and resource. However, it does not explicitly differentiate from sibling tools like grep or bash, though the action of reading a file is distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as bash for file reading or grep for searching. There is no mention of prerequisites, exclusions, or scenarios where another tool would be preferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_fileB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for behavioral disclosure. It mentions file creation on missing files but omits critical behavior such as overwriting existing files, path handling, or error conditions. This is a significant gap for a write operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The two-sentence description is concise and front-loaded with the primary action. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple but still has edge cases like overwrite and directory creation. The description covers only one aspect (creation) and lacks output expectations or permission requirements. With no output schema and no annotations, the description is incomplete for reliable use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does not clarify what 'file' refers to (path, name, format) or any constraints on 'content'. The description merely restates the parameter names without adding meaningful semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool writes content to a file and creates it if missing. The verb 'write' with the resource 'file' is specific and distinguishes it from siblings like read_file and delete_folder.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like bash, python_file, or create_folder. The description lacks context for typical use cases or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_folderB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It does disclose that a non-existent folder yields a message, but omits critical destructive details such as whether deletion is recursive, permanent, or what the success return value is. This is insufficient for a destructive tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the core action, using just two sentences. The second sentence covers an edge case without unnecessary words, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is a simple single-parameter delete operation, but the description omits the success response and whether deletion applies to non-empty folders. Without an output schema or annotations, these gaps leave the agent uncertain about the tool's full behavior. The non-existence message is helpful but not sufficient for complete context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'folder' is described as 'given name', adding some meaning beyond the raw schema string type. However, it does not clarify if this is a path or a simple name, leaving ambiguity. Since there is no schema description, this partial compensation warrants a mid score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it deletes a folder by name using the specific verb 'delete' and resource 'folder'. This inherently distinguishes it from sibling read/write/create tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like bash or python_file for deletion. It implies usage for folder deletion but lacks exclusions or context for choosing it over other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• globB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It does not mention pattern syntax (glob vs regex), recursive behavior, hidden files, case sensitivity, or return format. The description is too vague to set accurate expectations for an AI agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise at two sentences with no excessive detail. The first sentence states the core functionality, and the second adds a use case. Minor redundancy exists with 'This function' repeated, but it does not significantly impair clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no output schema or annotations, the description should explain pattern syntax, recursion behavior, and the nature of returned paths. These are all absent. The description is complete enough for a rough idea but too incomplete for reliable tool invocation in varied scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides no description for the pattern parameter (0% coverage), so the description must compensate. It only says 'match the given pattern,' which adds no meaningful detail about the required pattern syntax (e.g., wildcards, globbing rules). This is insufficient for an agent to construct a correct pattern.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a list of files matching a pattern, which is a specific verb+resource. It distinguishes itself from sibling tools like read_file, grep, and bash, none of which focus on listing files by pattern. The phrasing 'finding files in a directory' reinforces the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a use case ('useful for finding files in a directory') but does not explicitly state when to use this tool over alternatives, nor any exclusions or prerequisites. It implies usage for file discovery but lacks the explicit guidance seen in stronger examples.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_folderB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses an important behavioral trait: if the folder already exists, it returns a message rather than overwriting. However, it does not mention what happens upon success, error conditions (e.g., invalid path, permissions), or whether intermediate directories are created, leaving gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, concise and front-loaded with the core purpose. Every word adds value, and there is no redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (one parameter, no annotations, no output schema), the description covers the basic action and one edge case (existing folder), but it omits return format, success behavior, and parameter format. These omissions make it incomplete for reliable invocation, though the core idea is conveyed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description only says 'with the given name' referencing the 'folder' parameter. This does not clarify whether the parameter expects a full path, relative path, or just a folder name, so the agent lacks essential information to invoke the tool correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action: 'create a folder with the given name.' This specifies the verb (create) and resource (folder), and the existence of a sibling tool delete_folder differentiates it by opposite purpose. No ambiguity about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus alternatives like bash or python_code, which could also create folders. It only describes the function without context, exclusions, or mention of sibling tools, leaving the agent without usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.