Server Quality Checklist

Tool Scores

• convert_bug_to_testsA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate idempotent (idempotentHint=true) and non-destructive (destructiveHint=false). The description adds value by stating the tool returns 'root cause analysis, affected areas, and generated test cases with steps,' clarifying behavioral outcomes beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the core action and outputs, with no extraneous words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity (2 parameters, 100% schema coverage, no output schema), the description sufficiently explains inputs and outputs. It could mention any prerequisites (e.g., existence of a test harness) but is broadly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds semantics: it explains that 'severity' influences 'test priority ranking,' which is not in the schema description. This extra context helps the agent use the parameter effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('analyze' and 'generate') and resource ('bug report' and 'regression tests'), making the tool's purpose clear. It does not explicitly differentiate from siblings like 'build_regression_pack' or 'generate_test_scripts', but the focus on bug-to-test conversion is distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises providing a 'detailed bug description for best results,' which gives a usage hint but does not specify when to use this tool versus alternatives (e.g., when to use 'generate_test_scripts' instead) or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_pr_riskA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so description does not need to disclose safety. Adds output details (risk score, drivers, etc.) but no further behavioral traits like rate limits or auth needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states purpose, second adds usage and output highlights. No redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 parameters (all well-documented in schema) and no output schema, the description gives a solid overview of the return value and usage context. Could mention prerequisites like repository access, but overall complete enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so each parameter is already documented. Description adds no additional parameter meaning beyond what schema provides, but it does describe output structure which is not part of parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states 'Analyze a pull request for quality risk' with a specific verb and resource, and distinguishes from siblings like analyze_release_readiness and generate_missing_coverage by focusing on risk analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'Use before merging to catch regressions,' providing clear context. Does not mention when not to use or specific alternatives, but sibling list offers alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• build_regression_packA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate idempotentHint=true and destructiveHint=false. The description adds that the tool 'Returns a set of test cases', which is a fair behavioral note. However, it does not elaborate on permissions, rate limits, or other nuances. With annotations covering safety, a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no waste: purpose, usage guideline, and output. Front-loaded with the core action. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only 2 parameters, no output schema, and annotations present, the description is adequately complete. It explains the output as a 'set of test cases', which is sufficient. Could mention idempotency explicitly, but annotations cover it. Slightly vague on output format, but not a major gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters (area, context) are fully described in the input schema with 100% coverage. The description adds minimal extra meaning beyond repeating schema descriptions. Baseline 3 is correct as the schema handles most of the parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Generate' and the resource 'regression test pack'. It specifies the focus on a feature area or recent change, distinguishing it from sibling tools like 'generate_missing_coverage' or 'analyze_pr_risk' which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'after a refactor, migration, or risky change'. Provides clear context with the example. While it doesn't explicitly state when not to use or directly compare with siblings, the guidance is strong and sufficient for most agents.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_missing_coverageA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=false (mutation) and destructiveHint=false. The description adds that the tool submits a job and polls for results up to 2 minutes, which is key behavioral context beyond annotations. It does not cover failure handling or rate limits, but the polling timeout is valuable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that front-load the core purpose and then describe the process and output. Every word adds value; no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description informs about the return format (categorized tests with steps and expected results). It also mentions the polling timeout. This is fairly complete for a job-submission tool, though it could mention potential error states or job failure behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and each parameter already has a description. The tool description does not add new semantics beyond summarizing the overall functionality. Baseline 3 is appropriate as the schema already does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it generates test cases for stories/features needing coverage, specifies the job submission and polling process with a 2-minute timeout, and mentions the output format (categorized tests with steps and expected results). This is specific and distinct from sibling tools like generate_test_scripts or convert_bug_to_tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage 'for a user story or feature that needs better coverage,' but does not explicitly compare to sibling tools or provide when-not-to-use guidance. For example, it doesn't contrast with 'generate_test_scripts' or 'convert_bug_to_tests,' leaving the agent to infer the best context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_test_scriptsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate a non-readOnly and non-destructive operation. The description adds that it generates runnable automation code and mentions specific frameworks, but does not disclose rate limits, authentication needs, or details about side effects beyond the conversion.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with four sentences, front-loading the purpose. Every sentence adds value without redundancy or wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters, 1 required, and no output schema, the description adequately covers the workflow: source job, framework selection, and optional test case filtering. It omits return format but is sufficient for agent invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for all 4 parameters. The description adds minimal extra meaning, e.g., tying framework to the four options and mentioning the default. Since schema already does the heavy lifting, a 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool converts test cases into executable test scripts, specifies the required source_job_id, and lists supported frameworks. It distinguishes itself from siblings like generate_missing_coverage and convert_bug_to_tests by positioning itself as the second step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly requires a source_job_id from a prior generate_missing_coverage or convert_bug_to_tests call, and frames this as the second step. It does not explicitly exclude any scenarios or mention alternatives, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orangepro_get_agent_logsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so description's additional mention of 'timestamped log lines' adds minor value beyond annotations. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with verb and resource. Every sentence is informative and there is no waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With annotations and full schema coverage, the description adequately explains the tool's purpose and behavior. Lacks explicit details on pagination, but schema covers limit/offset. Sufficient for the complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and description does not add significant meaning beyond what the schema already provides for each parameter. Baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states verb ('Read'), resource ('log lines for an OrangePro agent'), and purpose with specific use cases (debug, check details, verify). Distinguishes from siblings like 'orangepro_get_agent' or 'orangepro_list_agent_runs'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'to debug failures, check processing details, or verify what an agent did during a run.' Does not provide explicit exclusions or compare to siblings, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orangepro_list_agent_runsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the description adds value by listing return fields (run_id, status, etc.). However, it does not disclose pagination behavior or ordering, which would be useful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with purpose, and no wasted words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description includes return fields. It covers the main usage but could mention default ordering (e.g., recent first) or pagination hints. Overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description does not add significant extra meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List recent runs for a specific OrangePro agent' with a specific verb and resource. It distinguishes from sibling tools like orangepro_list_agents (lists agents) and orangepro_run_agent (run agent).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases: 'check run history, find failed runs, or verify a recent run completed.' It does not explicitly mention when not to use or alternatives, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orangepro_list_agentsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, indicating a safe read operation. The description confirms this by stating it returns a list of agents with specific fields, but does not add behavioral context beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action and purpose, and contains no unnecessary words. It efficiently conveys the primary use case and return fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list tool with no output schema, the description compensates by listing the returned fields (agent_id, name, type, status, last run timestamp). It does not mention pagination or ordering, but for a simple discovery tool, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage for its single optional parameter (tenant_id) with a description. The tool description does not add additional parameter guidance beyond what is in the schema, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists all configured OrangePro agents for a tenant, and specifies the returned fields (agent_id, name, type, status, last run timestamp). It distinctly differentiates itself from sibling tools like orangepro_get_agent (single agent) and orangepro_list_agent_runs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises to use this tool first to discover available agents before getting details or triggering runs, which provides clear guidance on when to use it. It does not list when not to use it or alternatives, but the context from sibling names implies the workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orangepro_resolve_storyA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare idempotentHint=true, destructiveHint=false. The description adds that it returns entities and scores, but does not disclose additional traits like required permissions or rate limits. With annotations covering safety, a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loading the action and outcome. Every sentence is informative and there is no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters and no output schema, the description mentions return types (entities, concepts, confidence scores) which aids completeness. It could elaborate more on 'resolve' semantics or interpretation of scores, but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description provides no additional detail on parameters beyond what the schema already describes. No extra meaning is added.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'resolve' and resource 'OrangePro Knowledge Graph', clearly stating the tool's function. It lists return values (grounded entities, matched concepts, confidence scores) and allows it to be distinguished from sibling tools that deal with agents or test generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use to verify story coverage or find KG gaps', providing clear usage context. However, it does not mention when to avoid this tool or suggest alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_coverage_gapsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the tool as read-only and non-destructive. The description adds minor behavioral context (color coding, test counts) but does not need to go beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences convey purpose, output, and usage without waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional parameter and no output schema, the description fully covers what the tool does and what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides full description for the single parameter 'area' (case-insensitive filter). The tool description adds no further parameter details, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool finds application areas lacking test coverage and returns a heatmap with colored zones and test counts. It distinguishes from siblings like analyze_pr_risk or generate_missing_coverage.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Use to identify where to generate additional tests,' providing clear context. However, it does not mention when not to use or compare directly with sibling tools like generate_missing_coverage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_quality_riskA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=true and destructiveHint=false, so the agent knows this is safe. The description adds behavioral context: it uses coverage heatmap, execution history, and 30-day trend data to identify high- and medium-risk areas. This goes beyond annotations, though it does not detail data sources or outputs fully.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the tool's core action and data sources. Every sentence serves a purpose: the first explains what and how, the second gives usage examples. No extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description partially compensates by stating it identifies high- and medium-risk areas. Parameters are well-documented. However, it does not specify the output format (e.g., a list, scores, or visualization) or how the risk assessment is presented, which could leave the agent uncertain about return structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with both parameters described sufficiently in the input schema. The description adds example questions but does not elaborate on the 'area' parameter beyond what the schema says. It provides minimal additional semantic value over the schema, so a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a quality risk assessment using specific data sources (coverage heatmap, execution history, 30-day trend data) and identifies high/medium risk areas. Example questions like 'are we safe to ship?' further clarify purpose. The name 'explain_quality_risk' matches, and it is distinct from siblings like 'analyze_pr_risk' or 'analyze_release_readiness' which focus on different contexts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance via example questions ('are we safe to ship?', 'what areas need more tests?'), indicating when to use this tool. It does not explicitly state when not to use it or mention alternative tools, but the sibling list implies alternatives exist and the description sets a clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orangepro_get_agentA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the agent knows this is a safe read-only operation. The description adds that it returns 'agent config, run history, and current status,' which is helpful but does not go beyond what annotations guarantee. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, each serving a distinct purpose: first sentence defines the function, second sentence gives usage guidance and return summary. No filler words, front-loaded with the key action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is no output schema, the description adequately covers what the tool returns ('agent config, run history, and current status'). The tool is a single-agent getter, and the sibling tools cover more specialized queries, so this description is complete enough for its role.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with both agent_id and tenant_id described. The description adds value by clarifying that agent_id comes 'from orangepro_list_agents' and that tenant_id 'Defaults to ORANGEPRO_TENANT_ID env var,' which is useful context beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get full detail, configuration, timeline, and recent runs for a specific OrangePro agent.' It uses a specific verb ('Get') and a well-defined resource ('OrangePro agent'), and differentiates itself from sibling tools like orangepro_list_agents by focusing on inspecting a single agent in depth.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use after orangepro_list_agents to inspect a particular agent.' This provides clear context for when to use the tool. It does not explicitly state when not to use it or list alternatives, but the sibling tools (e.g., orangepro_get_agent_health, orangepro_get_agent_logs) suggest specialized uses, and the description implies this is a general inspection tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orangepro_get_agent_healthA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and destructiveHint. The description adds valuable context about what is checked (source config, auth, runtime status), surpassing the annotations. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. First sentence concisely states purpose, second offers usage guidance. Well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple health check tool with only 2 parameters and no output schema, the description fully covers purpose, usage, and behavior. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so parameters are well-documented. The description does not add parameter-level details but provides context for the tool's overall function. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Read' and resource 'health and connectivity status', and specifies it diagnoses agent failure by checking source config, auth, and runtime status. It effectively distinguishes from siblings like orangepro_get_agent or orangepro_get_agent_logs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'Use to diagnose why an agent is failing', providing a clear context. However, it does not explicitly mention when not to use or list alternatives, though sibling tools are available.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_release_readinessA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint=true, so the description properly confirms a read operation. It adds value by detailing the contents of the assessment (coverage analysis, execution summary, etc.), leaving no ambiguity about what the tool returns.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two efficient sentences: the first defines purpose and output, the second gives usage. No unnecessary words; content is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description enumerates all key return components (recommendation, confidence score, coverage analysis, etc.), fully informing the agent of what to expect. This compensates for the missing schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema coverage is 100%. No parameter information is needed, and the description adds nothing about parameters, which is appropriate. Baseline for 0 parameters is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get a tenant-wide release readiness assessment.' It lists specific outputs (recommendation, confidence score, coverage analysis, etc.), making it distinct from sibling tools like analyze_pr_risk which focus on individual PR risk.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Use before deciding whether to release.' This clearly indicates when to use the tool, though it does not explicitly mention when not to use it or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orangepro_run_agentA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description confirms the idempotentHint annotation by stating it's safe to retry and explains the API returns the active run if in progress, adding behavioral context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences, front-loaded with the action and safety note. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 2 parameters and no output schema, the description covers purpose, idempotency, common uses, and parameter origins. Nothing is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for both parameters. The description adds valuable context: agent_id is obtained from list_agents, tenant_id defaults to an environment variable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs ('Start'), identifies the resource ('OrangePro agent run'), and lists concrete use cases (data ingestion, KG sync, test generation). It clearly distinguishes from sibling tools like orangepro_list_agents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states it is safe to retry and provides example use cases. While it doesn't list when not to use it or mention alternatives, the context and sibling tool list imply appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.