Server Quality Checklist

Tool Scores

• heroku_config_varsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the tool can 'Get or set' config vars, implying both read and write operations, but doesn't specify permissions required, whether changes are reversible, rate limits, or what happens on success/failure (e.g., response format). This is a significant gap for a tool with mutation capabilities and no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core functionality ('Get or set config vars') and specifies the resource ('for a Heroku app'). There is zero waste or redundancy, making it appropriately sized and easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (handles both read and write operations with 4 parameters), lack of annotations, and no output schema, the description is incomplete. It doesn't cover behavioral aspects like mutation effects, error handling, or return values, leaving gaps that could hinder an AI agent's ability to use the tool correctly in varied contexts.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters (appName, set, key, value) with clear descriptions. The description adds no additional meaning beyond what the schema provides, such as explaining interactions between parameters (e.g., how 'set' toggles behavior) or usage examples. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Get or set config vars (environment variables) for a Heroku app,' which includes a specific verb ('Get or set') and resource ('config vars for a Heroku app'). It distinguishes from siblings like 'heroku_get_app' or 'heroku_list_apps' by focusing on environment variables, but doesn't explicitly differentiate from potential overlapping tools (e.g., if there were a 'heroku_get_config' sibling).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing the app name), when to prefer this over other config management tools, or any exclusions (e.g., not for app-level settings). Usage is implied through the parameter 'set' but not explicitly stated in the description text.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heroku_get_appC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states this is a read operation ('Get'), implying it's non-destructive, but doesn't cover critical aspects like authentication requirements, rate limits, error handling, or the format of returned data (e.g., JSON structure). For a tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Get detailed information about a Heroku app') and adds specific details ('including dynos and formation'). There is no wasted language, and it directly addresses what the tool does without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of retrieving app details, the lack of annotations and output schema means the description should do more to be complete. It doesn't explain what 'detailed information' entails beyond dynos and formation, how data is returned, or any behavioral traits like permissions. For a tool with no structured support, this description is insufficient to fully guide an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the single parameter 'appName' fully documented in the schema as 'Name of the Heroku app'. The description adds no additional meaning beyond this, such as format examples or constraints. Since the schema does the heavy lifting, the baseline score of 3 is appropriate, as the description doesn't compensate but doesn't detract either.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('detailed information about a Heroku app'), specifying what data is retrieved ('including dynos and formation'). It distinguishes from siblings like heroku_list_apps (which lists apps) and heroku_get_logs (which retrieves logs). However, it doesn't explicitly contrast with heroku_config_vars or heroku_list_addons, which also provide app-specific details, keeping it from a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing the app name), exclusions (e.g., not for modifying the app), or comparisons to siblings like heroku_list_apps (for listing apps) or heroku_config_vars (for configuration details). This leaves the agent to infer usage from context alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heroku_get_logsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions filtering capabilities but lacks critical details such as whether this is a read-only operation (implied but not stated), rate limits, authentication requirements, or what the output format looks like (e.g., structured vs. raw text). This leaves significant gaps for an agent to understand the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with just two sentences that directly convey the core functionality and filtering options. It's front-loaded with the main purpose and wastes no words, making it highly efficient for an agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, no output schema, no annotations), the description is incomplete. It lacks information on output format, error handling, or behavioral constraints like rate limits. While the schema covers inputs well, the description doesn't compensate for the absence of annotations or output schema, leaving the agent with insufficient context for reliable use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds some value by hinting at filtering parameters ('dyno or source'), but with 100% schema description coverage, all parameters (appName, lines, dyno, source) are already well-documented in the input schema. The description doesn't provide additional syntax, format details, or usage examples beyond what the schema offers, so it meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get recent logs') and resource ('from a Heroku app'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'heroku_get_app' or 'heroku_list_releases', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance by mentioning filtering options ('Can filter by dyno or source'), but it doesn't specify when to use this tool versus alternatives like 'heroku_get_app' for app details or 'heroku_list_releases' for release logs. No explicit when-not-to-use or prerequisite information is included.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heroku_list_releasesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool lists releases and shows deployment history, but doesn't disclose behavioral traits such as authentication requirements, rate limits, pagination (beyond the limit parameter), error handling, or the format of returned data. This is a significant gap for a read operation with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with zero waste. It front-loads the core action ('List releases for a Heroku app') and adds useful context ('Shows deployment history'). Every sentence earns its place, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (a read operation with 2 parameters), no annotations, and no output schema, the description is incomplete. It lacks details on authentication, rate limits, data format, error cases, and how it differs from siblings. For a tool with no structured support, the description should provide more context to be fully helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear descriptions for both parameters (appName and limit). The description adds no additional meaning beyond the schema, such as parameter interactions or examples. Since the schema does the heavy lifting, the baseline score of 3 is appropriate, as the description doesn't compensate but also doesn't detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('releases for a Heroku app'), and specifies the scope ('deployment history'). It distinguishes from siblings like heroku_list_apps (which lists apps) and heroku_get_logs (which retrieves logs), but doesn't explicitly differentiate from heroku_get_app (which gets app details). The purpose is specific but could be more distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an app name), exclusions, or comparisons to siblings like heroku_get_app (for app details) or heroku_list_addons (for add-ons). Usage is implied by the action but lacks explicit context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heroku_scaleC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool changes dyno quantities, implying a mutation operation, but doesn't cover critical aspects like required permissions, rate limits, cost implications, or what happens to existing dynos. This is a significant gap for a mutation tool without annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with zero waste, front-loaded with the core purpose. Every word earns its place, making it efficient and easy to parse without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (mutation with 4 parameters), lack of annotations, and no output schema, the description is incomplete. It doesn't address behavioral risks, output format, error conditions, or usage context, leaving gaps that could hinder an AI agent's ability to invoke it correctly and safely.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters (appName, dyno, quantity, size). The description adds no additional meaning beyond implying these parameters are used for scaling, but doesn't explain interactions (e.g., how size affects scaling) or provide examples. Baseline 3 is appropriate as the schema handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Scale dynos') and resource ('for a Heroku app'), with the specific verb 'Change the number of running dynos' adding detail. It distinguishes from siblings like heroku_config_vars or heroku_get_logs by focusing on scaling operations, though it doesn't explicitly differentiate from heroku_restart, which might also affect dyno state.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., app must exist), exclusions (e.g., cannot scale beyond limits), or compare to siblings like heroku_restart for dyno management. The description implies usage for scaling but lacks explicit context or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heroku_restartB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the action (restart) but doesn't mention important behavioral aspects: whether this causes downtime, what permissions are required, if there are rate limits, whether the restart is immediate or gradual, or what happens to in-process requests. For a potentially disruptive operation with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise - two sentences that directly communicate the tool's functionality and parameter behavior with zero wasted words. It's front-loaded with the core purpose and efficiently explains the parameter implication. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given this is a potentially disruptive operation (restarting dynos) with no annotations and no output schema, the description is incomplete. It doesn't address critical context like downtime implications, error conditions, permission requirements, or what success/failure looks like. For a mutation tool in a production environment, more behavioral context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds minimal value beyond the schema by explaining that omitting 'dyno' restarts all dynos, which is already implied by the schema's 'Omit to restart all' text. No additional semantic context is provided beyond what's in the structured schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('restart dynos') and resource ('Heroku app'), making the purpose immediately understandable. It distinguishes between restarting all dynos vs specific ones, though it doesn't explicitly differentiate from sibling tools like heroku_scale (which also manages dynos). The description is specific but lacks sibling comparison context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you need to restart dynos) and provides basic parameter guidance (omit dyno to restart all). However, it doesn't explicitly state when to choose this tool over alternatives like heroku_scale for dyno management, nor does it mention prerequisites or potential side effects. The guidance is functional but incomplete.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heroku_list_appsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions optional team filtering but doesn't describe return format (e.g., list structure, pagination), authentication needs, rate limits, or whether this is a read-only operation. For a listing tool with zero annotation coverage, this leaves significant gaps in understanding how the tool behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise - two short sentences that communicate the core functionality and optional feature without any wasted words. It's front-loaded with the primary purpose and efficiently adds the filtering detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (single optional parameter) and high schema coverage, the description is minimally adequate. However, with no annotations and no output schema, it should ideally provide more behavioral context about what the list contains, authentication requirements, or response format. The description meets basic needs but leaves room for improvement.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single optional 'team' parameter completely. The description adds marginal value by mentioning filtering capability but doesn't provide additional context beyond what's in the schema (e.g., how team names are formatted, what happens without the parameter). Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and resource 'Heroku apps', making the purpose immediately understandable. It distinguishes from siblings like heroku_get_app (single app) and heroku_list_addons/releases (different resources). However, it doesn't explicitly contrast with heroku_config_vars or heroku_get_logs which might also involve listing operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by mentioning optional team filtering, suggesting this tool is for retrieving app inventories. However, it provides no explicit guidance on when to use this versus heroku_get_app (single app) or heroku_list_addons/releases (different resources), nor does it mention prerequisites like authentication requirements.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heroku_list_addonsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states it's a list operation, implying read-only behavior, but doesn't cover aspects like pagination, rate limits, error conditions, or authentication requirements. It's minimal but not contradictory.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('List all add-ons attached to a Heroku app') and adds clarifying examples ('databases, caches, etc.') without any wasted words. It's appropriately sized for this simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema, no annotations), the description is adequate but not complete. It covers the basic purpose but lacks behavioral details like response format or error handling, which would be helpful for an agent despite the simple context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'appName' fully documented in the schema. The description doesn't add any extra meaning about the parameter beyond what's in the schema, so it meets the baseline of 3 where the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all add-ons attached to a Heroku app'), with specific examples ('databases, caches, etc.') that help distinguish it from sibling tools like heroku_list_apps or heroku_list_releases. It precisely defines scope without being tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you need to see add-ons for a specific app, but it doesn't explicitly state when to use this versus alternatives like heroku_get_app (which might provide app details) or heroku_config_vars (which could relate to add-on configuration). No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.