Server Quality Checklist

Tool Scores

• deploy_projectC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the deployment action and file handling details, implying a write/mutation operation, but fails to disclose critical traits: whether deployment is reversible, what permissions are required, if there are rate limits, or what happens on success/failure (e.g., URL generation, error handling). For a mutation tool with zero annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by specific implementation details. Both sentences earn their place by clarifying the action and file handling. It's appropriately sized for a 2-parameter tool, though it could be slightly more structured (e.g., bullet points for file rules).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (deployment mutation with file handling), lack of annotations, and no output schema, the description is incomplete. It doesn't explain return values (e.g., success status, generated URL), error conditions, or side effects. For a tool that likely alters system state and involves binary data, more context is needed to be adequately helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('project_id' and 'files') thoroughly. The description adds marginal value by reiterating the file map structure and binary file prefix ('__b64__'), but doesn't provide additional syntax, format details, or examples beyond what the schema states. This meets the baseline of 3 when schema coverage is high.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Deploy the project') and target ('to a live branded URL'), specifying the resource (project) and outcome (live URL). It distinguishes from siblings like 'create_project' or 'write_file' by focusing on deployment rather than creation or file operations. However, it doesn't explicitly differentiate from potential deployment-related siblings (none listed), keeping it at 4 instead of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'create_project' for initial setup or 'run_command' for other deployment methods. It mentions file handling specifics but doesn't clarify prerequisites (e.g., must have a project created first) or exclusions (e.g., not for testing). This lack of contextual usage advice results in a low score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_projectA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool creates something new and returns specific data (project ID and URL ID), which is helpful. However, it doesn't mention permissions needed, whether creation is idempotent, rate limits, or error conditions - leaving significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise - just two sentences that directly state the tool's purpose and return values. Every word earns its place with zero redundancy or unnecessary elaboration. It's front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a creation tool with no annotations and no output schema, the description provides basic purpose and return information but lacks important context. It doesn't explain what happens after creation, error handling, or how the returned IDs should be used. The minimal information is adequate but leaves clear gaps for a mutation operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both parameters. The description adds no additional parameter information beyond what's in the schema. This meets the baseline expectation when schema coverage is complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Create a new AICre8 project') and resource ('project'), distinguishing it from siblings like list_projects or deploy_project. It's not a tautology of the name and provides concrete information about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, when not to use it, or compare it to sibling tools like list_projects or deploy_project. The agent must infer usage from context alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_commandA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds context about the timeout constraint (5 minutes max) and the sandbox environment, which is useful. However, it doesn't cover critical aspects like security implications, error handling, output format, or whether commands are executed with specific permissions, leaving significant gaps for a tool that executes arbitrary shell commands.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise with just two sentences that directly convey the core functionality and a key constraint. Every word earns its place, and it's front-loaded with the main purpose, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of executing arbitrary shell commands (a high-risk operation), no annotations, and no output schema, the description is incomplete. It covers the basic purpose and timeout but lacks details on security, error handling, return values, or dependencies on other tools like 'create_project'. For such a tool, more context is needed to ensure safe and effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 4 parameters thoroughly. The description doesn't add any parameter-specific details beyond what's in the schema (e.g., it doesn't explain 'project_id' or 'cwd' further). Baseline 3 is appropriate when the schema does the heavy lifting, though no extra value is provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'execute' and resource 'shell command in the project sandbox', with examples like 'npm install, npm run build' that make the purpose specific and understandable. However, it doesn't explicitly differentiate from sibling tools like 'deploy_project' or 'generate_code', which might also involve command execution in some contexts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for executing shell commands in a project sandbox, but doesn't provide explicit guidance on when to use this versus alternatives like 'deploy_project' or 'generate_code'. It mentions a 'max timeout 5 minutes', which offers some operational context but no clear when-not-to-use or prerequisite information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_projectsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It indicates a read operation ('List') and specifies the returned fields, but lacks details on behavioral traits such as pagination, rate limits, authentication needs, or error handling. It doesn't contradict annotations, but offers minimal behavioral context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action ('List all your AICre8 projects') and adds value by specifying the returned data. There is zero waste, and every part of the sentence contributes to understanding the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters, no output schema, no annotations), the description is adequate but has gaps. It explains what the tool does and what it returns, but without annotations or output schema, it misses details like response format, error cases, or usage constraints, making it minimally viable but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately adds no parameter information, focusing instead on the tool's purpose and output details, which is sufficient given the lack of parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List all your AICre8 projects') and specifies what information is returned ('with their IDs, names, and preview URLs'), which provides a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'create_project' or 'deploy_project' in terms of when to choose listing versus creation/deployment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying 'List all your AICre8 projects,' suggesting it's for retrieving existing projects, but it doesn't provide explicit guidance on when to use this tool versus alternatives like 'create_project' or 'deploy_project.' No exclusions or clear context for alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_fileA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It effectively describes key behaviors: it performs write/update operations (implying mutation), automatically creates parent directories (a helpful implementation detail), and operates within a sandbox environment (suggesting isolation). However, it doesn't cover permissions, error conditions, or response format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two sentences) with zero wasted words. The first sentence states the core purpose, and the second adds valuable behavioral context. Every element earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description provides adequate basic information about what the tool does and its automatic directory creation behavior. However, it lacks details about error handling, response format, or specific constraints that would be helpful for complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters thoroughly. The description adds no additional parameter information beyond what's in the schema, maintaining the baseline score of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('write or update a file'), the target resource ('in the project sandbox'), and includes a behavioral detail ('creates parent directories automatically'). It distinguishes itself from sibling tools like 'read_file' by specifying a write operation rather than a read operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for file writing/updating in a project context, but provides no explicit guidance on when to use this tool versus alternatives like 'generate_code' or 'run_command'. It mentions the sandbox environment but doesn't clarify prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_codeA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and adds valuable behavioral context beyond basic functionality: it discloses that the tool 'will write files, install packages, and update the project' and includes cost information ('Costs 1 credit per call'), though it lacks details on error handling or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with two concise sentences that efficiently convey the tool's purpose, actions, and cost without any wasted words, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (AI-driven code generation with side effects) and no annotations or output schema, the description is mostly complete: it covers purpose, actions, and cost. However, it lacks details on return values or error conditions, which could help an agent handle responses better.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters. The description does not add any parameter-specific details beyond what the schema provides, such as examples for the prompt or project_id formats, resulting in a baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('generate or modify code') and resources ('AICre8 project using AI'), distinguishing it from siblings like create_project or write_file by focusing on AI-driven code generation rather than manual file operations or project management.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('Send a prompt to generate or modify code') and mentions a cost implication ('Costs 1 credit per call'), but it does not explicitly state when not to use it or name alternatives like write_file for manual edits, leaving some guidance gaps.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_fileA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: it's a read operation (implied by 'Read'), requires an active sandbox (a prerequisite condition), and hints at context (sandbox environment). However, it doesn't mention potential errors, rate limits, or output format details, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence and adds a crucial prerequisite in the second. Every sentence earns its place with no wasted words, making it highly efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (read operation with prerequisites), no annotations, and no output schema, the description is mostly complete. It covers purpose, usage, and prerequisites but lacks details on return values or error handling, which would be beneficial for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all parameters. The description adds no additional parameter semantics beyond what the schema provides, such as examples or usage notes. Baseline 3 is appropriate as the schema handles the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Read a file') and resource ('from the project sandbox'), distinguishing it from siblings like write_file (which writes) or list_projects (which lists). It precisely defines the tool's function without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use this tool ('Read a file from the project sandbox') and provides a prerequisite ('Requires an active sandbox') with a clear alternative action ('run generate_code first to boot it'), guiding the agent on proper usage versus other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.