Server Quality Checklist

Tool Scores

• deploy_heim_application_to_cloudB
  3.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate this is a non-read-only, non-destructive, idempotent operation with open-world semantics. The description adds valuable context by specifying it runs a command-line operation ('heim deploy') and that the deployment makes the application available on a path outputted in the console, which clarifies the execution method and output behavior beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences that directly state the action and outcome. It avoids unnecessary details, though the phrasing 'availible' contains a typo and could be slightly clearer (e.g., specifying 'available' and elaborating on the path output).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (deployment operation), annotations cover safety and idempotency well, and the schema fully describes the single parameter. However, without an output schema, the description only briefly mentions a path output in the console, lacking details on return format, error handling, or deployment status, leaving gaps for agent invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting the 'path' parameter as an absolute Windows path to the application folder root. The description doesn't add any parameter-specific details beyond what the schema provides, so it meets the baseline for high schema coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('deploy an application to Heim cloud') and resource ('application'), specifying it runs the 'heim deploy' command. However, it doesn't explicitly differentiate from sibling tools like 'deploy_heim_application' (which may be a local deployment variant), leaving some ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'deploy_heim_application' or 'heim_update'. The description mentions making the application available on a path, but doesn't specify prerequisites, exclusions, or comparative contexts with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heim_startB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations cover key traits: readOnlyHint=false, destructiveHint=false, idempotentHint=false, and openWorldHint=false. The description adds value by explaining the runtime's purpose ('run your backend applications'), but doesn't disclose additional behavioral details like startup time, error conditions, or dependencies. No contradiction with annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without redundancy. It's front-loaded with the core action and resource, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (starting a runtime) and lack of output schema, the description is minimal but adequate. Annotations provide safety and idempotency info, but the description could better address context like prerequisites or effects. It meets minimum viability but has gaps in usage guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately avoids discussing parameters, focusing on the tool's action. Baseline is 4 for zero parameters, as it doesn't add unnecessary details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Starts') and the target ('the Heim runtime'), specifying it will 'run your backend applications.' This provides a specific verb+resource combination, though it doesn't explicitly differentiate from sibling tools like 'heim_update' or 'new_heim_application' in terms of when to use each.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., whether the runtime must be stopped first), exclusions, or comparisons to siblings like 'heim_clear' or 'deploy_heim_application,' leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heim_updateB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate this is a mutable operation (readOnlyHint: false) but not destructive or idempotent. The description adds that it updates to the latest version, providing some behavioral context beyond annotations. However, it doesn't detail side effects, error conditions, or what 'latest version' entails (e.g., automatic checks, user confirmation).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It's front-loaded with the core action and target, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 0 parameters and annotations covering mutability and safety, the description is minimally adequate. However, without an output schema, it doesn't explain what happens after the update (e.g., success message, version info, errors), and it lacks context about sibling tools, leaving gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are 0 parameters, and schema description coverage is 100%, so no parameter documentation is needed. The description appropriately omits parameter details, earning a baseline score of 4 for not adding unnecessary information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Updates') and target ('Heim'), specifying it updates to the latest version. However, it doesn't differentiate from sibling tools like 'deploy_heim_application' or 'heim_start', which might also involve updates or deployments, leaving some ambiguity about when to use this specific update tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'deploy_heim_application' or 'heim_start'. The description lacks context about prerequisites, timing, or exclusions, leaving the agent to infer usage based on tool names alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heim_clearA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide key behavioral hints (destructiveHint: true, readOnlyHint: false, etc.), but the description adds valuable context beyond this: it specifies that clearing the cache 'will attempt to stop the runtime' and removes 'all applications, logs, metrics, etc.', clarifying the scope and impact of the destructive operation. No contradiction with annotations is present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the tool's purpose and effects without redundancy. It is front-loaded with the core action ('Clears the local cache') and adds necessary details, though it could be slightly more concise by avoiding 'etc.'

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (destructive operation with no parameters) and rich annotations, the description covers the core action and scope adequately. However, without an output schema, it lacks details on return values or success/failure indicators, and it doesn't address potential side effects like downtime or recovery steps, leaving some gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the baseline is 4. The description appropriately does not discuss parameters, as none exist, and focuses on the tool's action and effects, which is sufficient given the empty input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('clears') and resource ('local cache of the Heim runtime'), with explicit details about what gets removed ('all applications, logs, metrics, etc.'). It distinguishes from siblings like deploy_heim_application or heim_start by focusing on cleanup rather than deployment or startup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for clearing cache and stopping the runtime, but provides no explicit guidance on when to use this tool versus alternatives (e.g., heim_update for updates or heim_start for restarting). No prerequisites or exclusions are mentioned, leaving the agent with minimal context for decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• new_heim_applicationA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate this is not read-only, not open-world, not idempotent, and not destructive. The description adds useful context about creating a project folder and not modifying the heim folder, which goes beyond annotations. However, it doesn't mention potential side effects like file system changes or error conditions, leaving some behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the core purpose without unnecessary words. It's front-loaded with the main action and resource, making it easy to understand at a glance. Every word serves a clear purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 7 parameters, 100% schema coverage, and no output schema, the description provides sufficient context about what the tool does and its basic behavior. It could be more complete by mentioning the tool's effect on the local file system or potential errors, but it adequately covers the main purpose given the rich schema support.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the schema fully documents all 7 parameters. The description doesn't add any parameter-specific details beyond what's in the schema. The baseline score of 3 reflects adequate coverage through the schema alone, with no additional value from the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Runs heim new command'), the resource ('create application scaffolding'), and the source ('from an OpenAPI 3.0.1 specification'). It distinguishes itself from siblings like deploy_heim_application and heim_update by focusing on initial creation rather than deployment or updating.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for initial application creation from an OpenAPI spec, suggesting usage when starting a new project. However, it doesn't explicitly state when to use this versus alternatives like heim_update for existing projects, nor does it mention prerequisites or exclusions. The context is clear but lacks explicit guidance on alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deploy_heim_applicationA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable behavioral context beyond annotations: it specifies the deployment makes the application available on a specific local URL (http://127.0.0.1:3000<PATH>), which is not covered by annotations. Annotations already indicate it's non-destructive, non-read-only, and idempotent, so the description appropriately focuses on runtime behavior rather than repeating those traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first states the core action and target, the second explains the outcome (URL availability). Every phrase adds value without redundancy, making it front-loaded and appropriately concise for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (single parameter, no output schema), the description covers key aspects: purpose, local vs. cloud distinction, and deployment outcome. However, it lacks details on prerequisites (e.g., required OpenAPI spec), error conditions, or interaction with sibling tools like 'heim_clear', leaving minor gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema fully documents the single parameter ('path'). The description mentions '<PATH>' in the URL context but doesn't add meaningful semantics beyond what the schema provides (e.g., format examples or constraints). This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Runs 'heim deploy' command'), the resource ('deploy an application'), and the target environment ('to a local Heim runtime'). It distinguishes from sibling 'deploy_heim_application_to_cloud' by specifying 'local' deployment, making the purpose unambiguous and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (deploying to local runtime) and implicitly distinguishes it from the cloud deployment sibling. However, it lacks explicit guidance on when NOT to use it or alternatives for other scenarios (e.g., vs. 'heim_start' or 'new_heim_application'), which prevents a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.