Server Quality Checklist

Tool Scores

• validate_irC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It only mentions 'validate locally' and 'envelope + every declarative step' but omits details on what validation entails, side effects, or output behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, but it is underspecified and lacks structure. It sacrifices necessary details for brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, output schema, and parameter descriptions, the description is insufficient for an agent to correctly invoke the tool. It fails to provide parameter guidance and behavioral context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'app_ir' is an object with no description in the schema (0% coverage). The description does not explain what this parameter should contain, leaving the agent with only the type information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Validate' and the resource 'app.ir.json', with additional scope clarification 'locally (envelope + every declarative step)'. It effectively distinguishes from sibling tools like deploy_ir and smoke_ir.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies local validation before deployment but does not explicitly state when to use this tool versus alternatives like deploy_ir or smoke_ir. No when-not or alternative guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_read_toolB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the behavioral trait of refusing write/destructive actions, but with no annotations, it lacks details on what happens if a write is attempted, rate limits, or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short but fails to provide essential information about parameters and usage, making it under-specified rather than concise. It is front-loaded but incomplete.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has three parameters, no annotations, and no output schema, the description is incomplete. It does not explain what 'args' is for, what 'function' values are valid, or how to find them.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no meaning to any of the three parameters (app_id, function, args). The description does not explain what these parameters are or how to use them.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs a READ-only tool of a deployed app, specifying the verb 'run' and resource 'READ-only tool'. It distinguishes from sibling tools that likely involve writing or destructive actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (for read operations) and when not (refuses write/destructive). It does not explicitly name alternatives but the sibling tools suggest other operations (deploy, smoke, validate).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_appsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden of behavioral disclosure. It only states the action and the PII-masking behavior, lacking details on read-only nature, authentication requirements, rate limits, or potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the key information. Every word is necessary and there is no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has low complexity with no parameters and no output schema. The description provides the basic function but does not specify the output fields or any sorting/filtering behavior. While adequate for a simple list, it could be more complete to reduce ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, and the schema is empty with full coverage. The description adds value by noting that the output is PII-masked, which is not indicated by the schema, so it provides meaningful context beyond the structured data.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' and identifies the resource as 'caller's developer apps' with the qualifier 'PII-masked'. It clearly distinguishes this from sibling tools which focus on different operations like deploying or validating IR.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description does not mention prerequisites, exclusions, or any context about appropriate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deploy_irA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description partially discloses behavior by noting it creates the app record if needed. However, it omits other behavioral traits such as whether it updates existing apps, permissions required, or potential destructive actions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that conveys the core action and side-effect without unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (2 parameters, nested object, no output schema), the description covers the basic intent but lacks parameter details, return value info, and error handling guidance. It is minimally adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage for parameters, and the description only references 'app.ir.json' loosely related to app_ir. It does not explain what app_ir is or the role of app_id, leaving the agent without semantic guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'Deploy an app.ir.json' and specifies the target 'into the caller's account' with a side-effect note. This distinguishes it from sibling tools like get_app, list_apps, validate_ir, etc.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for deployment but does not explicitly state when to use or avoid this tool versus alternatives. No exclusion criteria or alternative guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• smoke_irA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must cover behavior. It mentions 'ISOLATED store' indicating safety from side effects, and output format {ok,result,trace}. However, it lacks details on auth requirements, rate limits, or what happens to the store after execution.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no fluff. Core information (verb, resource, isolation, output) is front-loaded. Each word serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description helpfully includes output format. But with nested objects and no parameter details, the description leaves gaps. The isolated store hint is valuable but overall could be more comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It does not explain the app_ir object structure, function name format, or how args should be provided. The description uses general terms 'one function' without clarifying parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a function from an app.ir.json in an isolated store and reports the result in a structured format. It distinguishes itself from siblings like deploy_ir (deployment) and validate_ir (validation) by focusing on isolated execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for testing functions in an isolated environment but does not explicitly state when to use or avoid this tool compared to alternatives like run_read_tool or deploy_ir. No when-not-to-use or prerequisite info.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_appA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the response is PII-masked, which is a behavioral trait. However, it does not explicitly state that the tool is read-only or address authentication/rate limits. For a simple get tool, this is adequate but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 6 words, no wasted words. It conveys the key purpose and the notable PII-masking feature concisely.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with 1 parameter and no output schema or annotations, the description is minimal. It covers the basic purpose and return content but lacks guidance on parameter semantics and error cases. It is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no parameter description in the description). The single parameter app_id is not explained beyond its name and type. With low coverage, the description should add meaning (e.g., how to obtain app_id), but it does not, leaving the agent to infer from the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'one app', and specifies what is returned (manifest, tools with action_type, PII-masked). This distinguishes it from sibling tools like list_apps (which lists all apps) and run_read_tool (which runs a tool).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: when you need details of a specific app by app_id. However, it does not provide explicit guidance on when to use this vs alternatives (e.g., list_apps), nor does it mention prerequisites or when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.