Server Quality Checklist

Tool Scores

• gateway_get_statsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While it indicates this is a read operation ('Get'), it doesn't mention authentication requirements, rate limits, whether this is real-time or historical data, or what format the statistics are returned in. For a statistics tool with zero annotation coverage, this leaves significant behavioral questions unanswered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that clearly states the tool's purpose. Every word earns its place by specifying what statistics are included, making it appropriately concise and front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a statistics tool with no annotations and no output schema, the description is insufficiently complete. It doesn't explain what format the statistics are returned in, whether they're aggregated or detailed, what time periods they cover, or how to interpret the results. The description should provide more context about the statistical output given the lack of structured documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions 'token savings' which relates to the 'price_per_million' parameter in the schema, but doesn't add meaningful semantic context beyond what the schema already provides (100% coverage). The schema already documents this parameter with a clear description and default value, so the description adds minimal additional value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('Gateway Statistics'), and specifies the types of statistics included (invocations, cache hits, token savings, top tools). However, it doesn't explicitly differentiate from sibling tools like 'gateway_cost_report' or 'gateway_get_profile' which might also provide statistical data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With multiple sibling tools that might provide related statistics (gateway_cost_report, gateway_get_profile), there's no indication of when this specific statistics tool is appropriate versus those other options.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_run_playbookC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden but only states it executes and collapses multiple tool calls. It lacks details on permissions needed, side effects (e.g., if it's destructive), error handling, or performance implications like rate limits, leaving significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly conveys the core functionality without any wasted words, making it easy to understand at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of executing multi-step playbooks, no annotations, and no output schema, the description is insufficient. It doesn't explain what a playbook is, expected outcomes, error cases, or integration with other tools, leaving critical context missing for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema documents parameters adequately. The description adds no additional meaning beyond the schema, such as examples of playbook names or argument structures, but doesn't contradict it, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('execute') and resource ('multi-step playbook'), and distinguishes it from siblings by specifying it collapses multiple tool calls into one invocation. However, it doesn't explicitly differentiate from 'gateway_invoke' which might also execute something, leaving some ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'gateway_invoke' or other siblings. The description implies it's for multi-step operations but doesn't specify prerequisites, exclusions, or typical scenarios for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_get_profileB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool 'shows' information, implying a read-only operation, but doesn't clarify if it requires authentication, has rate limits, or what happens if no profile is active. For a tool with zero annotation coverage, this is a significant gap in transparency, as it lacks details on permissions, errors, or response format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('show the active routing profile') and adds useful context ('for this session and what it allows or denies'). There is no wasted verbiage, and every word contributes to understanding the tool's function, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters, no output schema, no annotations), the description is minimally adequate. It explains what the tool does but lacks details on behavioral aspects like error handling or return values. Without an output schema, the description should ideally hint at the response format, but it doesn't, leaving some gaps in completeness for a tool that interacts with session settings.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the input schema has 100% description coverage (though empty). The description adds no parameter information, which is appropriate here. In such cases, the baseline score is 4, as there are no parameters to document, and the description doesn't need to compensate for any schema gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to 'show the active routing profile for this session and what it allows or denies.' It specifies the verb ('show') and resource ('active routing profile'), and distinguishes it from siblings like 'gateway_list_profiles' (which likely lists all profiles) and 'gateway_set_profile' (which modifies profiles). However, it doesn't explicitly contrast with these siblings, keeping it from a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by mentioning 'for this session,' suggesting it's used to check current session settings. However, it provides no explicit guidance on when to use this tool versus alternatives like 'gateway_list_profiles' or 'gateway_set_profile,' nor does it mention prerequisites or exclusions. This leaves usage somewhat inferred rather than clearly defined.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_set_profileA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions that routing profiles restrict tool/backend availability, which is helpful, but lacks details on permissions needed, whether changes are session-specific or persistent, error conditions, or side effects. For a mutation tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero waste: the first states the action and resource, and the second explains the purpose of routing profiles. It is front-loaded and efficiently structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (mutates session state), lack of annotations, and no output schema, the description is incomplete. It covers the basic purpose but misses behavioral details like effects, permissions, or response format. However, it does provide enough to understand the core function.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'profile' parameter with examples. The description does not add any parameter-specific details beyond what the schema provides, such as valid profile names or constraints, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Switch') and resource ('active routing profile for this session'), and distinguishes from siblings by focusing on profile activation rather than listing, getting, or other operations. It explains what a routing profile does, adding useful context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when needing to change tool/backend availability, but does not explicitly state when to use this vs. alternatives like 'gateway_get_profile' or 'gateway_list_profiles'. No exclusions or prerequisites are mentioned, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_webhook_statusA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It implies a read-only operation by using 'List', but lacks details on permissions, rate limits, pagination, or response format, leaving significant gaps for a tool that likely returns structured data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the purpose ('List registered webhook endpoints') and adds necessary detail ('delivery statistics') without any wasted words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a read operation with no parameters) and lack of annotations or output schema, the description is minimally adequate. It states what the tool does but omits behavioral details like response structure or constraints, making it incomplete for full agent understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema fully documents the absence of inputs. The description adds no parameter information, which is acceptable here, but a baseline of 4 is appropriate as it doesn't need to compensate for any gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List registered webhook endpoints') and the resources involved ('delivery statistics'), distinguishing it from siblings like gateway_get_stats or gateway_list_servers by focusing on webhook-specific data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like gateway_get_stats or gateway_list_servers, nor does it mention prerequisites or context for retrieving webhook status versus other gateway information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_revive_serverA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the core action (re-enabling routing) and an important side effect (resetting error budget), but doesn't mention potential consequences, permissions required, rate limits, or what happens if the server isn't actually disabled. It provides basic behavioral context but lacks completeness for a mutation operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that each add distinct value: the first states the primary action, the second reveals important behavioral context. There's zero wasted language, and the information is front-loaded with the core purpose immediately clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description provides adequate basic information about what the tool does but lacks details about what happens after invocation (success/failure indicators, return values, side effects beyond error budget reset). Given the complexity of server management operations, more context about dependencies, permissions, or system state requirements would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the schema already documents the single parameter thoroughly. The description doesn't add any parameter-specific information beyond what's in the schema, but since there's only one parameter and the schema coverage is complete, this represents minimal information loss. The baseline for high schema coverage would be 3, but the single parameter case with complete documentation justifies a slightly higher score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('re-enable routing') on a specific resource ('previously disabled backend server'), and distinguishes it from siblings by mentioning the unique 'reset error budget' functionality. It uses precise verbs and identifies the target resource without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context ('previously disabled backend server') but doesn't explicitly state when to use this tool versus alternatives like 'gateway_kill_server' or 'gateway_reload_config'. No guidance is provided about prerequisites, dependencies, or exclusions for using this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_cost_reportA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses behavioral traits such as the types of data returned (cost, call count, breakdowns) and time windows, but lacks details on permissions, rate limits, or error handling. It adequately describes the operation but could be more comprehensive for a tool with potential admin functions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by specific details in a compact format. Every sentence adds essential information without redundancy, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (cost reporting with admin options) and lack of annotations or output schema, the description is mostly complete. It covers what data is returned and parameter context, but could benefit from mentioning output format or error cases to be fully comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents parameters. The description adds value by explaining the context of the parameters (e.g., 'admin view' for include_all_keys and include_all_sessions, and default behavior for session_id), enhancing understanding beyond the schema's technical details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Return', 'Includes') and resources ('current session and API-key spend', 'total cost, call count, breakdown by backend and tool', 'Per-key totals'). It distinguishes itself from siblings by focusing on cost reporting, unlike tools like gateway_get_stats (general stats) or gateway_list_tools (tool listing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for monitoring spend and provides context on what data is included (e.g., 24h/7d/30d windows), but it does not explicitly state when to use this tool versus alternatives like gateway_get_stats or provide exclusions. Usage is inferred from the content rather than directly guided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_kill_serverA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: the immediate effect ('Immediately disable'), the partial nature of the disablement ('tools remain visible... but are marked as disabled'), and the operational context ('operator kill switch'). It doesn't mention authentication requirements, rate limits, or error conditions, but provides substantial behavioral context for a destructive operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that are perfectly front-loaded and efficient. The first sentence states the core action and context, the second clarifies the partial nature of the disablement. Every word earns its place with zero redundancy or wasted verbiage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive operation with no annotations and no output schema, the description provides good coverage of what the tool does and its behavioral effects. It explains the partial disablement (tools remain visible but marked disabled) which is crucial context. It doesn't describe the return value or error conditions, but given the tool's relative simplicity and clear purpose, the description is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description doesn't add any parameter-specific information beyond what's already in the schema (which has 100% coverage). The schema already documents that 'server' is a required string parameter representing the backend server name. No additional semantics about parameter format, constraints, or examples are provided in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Immediately disable routing') and target resource ('backend server'), distinguishing it from sibling tools like gateway_revive_server (which presumably re-enables servers) and gateway_list_servers (which only lists them). The 'operator kill switch' phrase reinforces the emergency nature of the operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('Immediately disable routing'), but doesn't explicitly state when NOT to use it or name specific alternatives. However, the existence of gateway_revive_server as a sibling implies this is for disabling rather than permanent removal, and the 'operator kill switch' wording suggests emergency scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_list_toolsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, non-destructive, and idempotent behavior, so the description doesn't need to repeat these. It adds useful context about the return format ('names and descriptions') and the scope of listing, but doesn't disclose additional behavioral traits like rate limits or pagination. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero waste: the first sentence states the purpose and parameter usage, and the second provides a clear alternative. It's front-loaded and efficiently structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one optional parameter), rich annotations covering safety and behavior, and no output schema, the description is mostly complete. It clarifies the return format and sibling differentiation, but could mention if the list is paginated or sorted. Still, it's adequate for the context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents the single parameter. The description adds no extra parameter details beyond what's in the schema, but it reinforces the semantics by explaining the effect of omitting the server parameter. Baseline 3 is appropriate as the schema handles the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('tools'), specifies the scope ('from a specific backend' or 'all tools across all backends'), and distinguishes it from sibling gateway_search_tools by noting the latter provides 'ranked results with full schemas'. This is specific and avoids tautology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use this tool (list tools with names and descriptions) versus an alternative (gateway_search_tools for ranked results with full schemas). The guidance on omitting the server parameter to list all tools provides clear context for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_invokeA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable behavioral context beyond annotations by specifying that it 'Routes through the gateway's auth, rate-limit, caching, and failsafe middleware.' While annotations provide readOnlyHint=false and openWorldHint=true, the description elaborates on the gateway's processing pipeline, which helps the agent understand the execution environment and potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that each earn their place: the first states the core functionality and behavioral traits, the second provides crucial usage guidance. No wasted words, front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (invoking arbitrary tools with middleware routing) and lack of output schema, the description does well by explaining the gateway middleware and discovery prerequisite. However, it doesn't detail response formats, error handling, or specific auth/rate-limit implications, leaving some gaps for a tool with openWorldHint=true and no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the schema already documents all three parameters (arguments, server, tool) adequately. The description doesn't add specific meaning about parameter usage beyond implying that 'server' and 'tool' should be discovered via gateway_search_tools, which is minimal additional value. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Invoke any tool on any backend server') and distinguishes it from siblings by mentioning routing through gateway middleware and the need to use gateway_search_tools first. It explicitly names the resource (tools on backend servers) and the verb (invoke) with clear scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Use gateway_search_tools first to discover the right tool and server') and implies when not to use it (when you already know the specific tool from other sibling tools). It names an alternative (gateway_search_tools) for discovery, making usage context very clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_list_profilesA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes the tool's function (listing profiles) and hints at context (profiles narrow the visible toolset), but lacks details on output format, pagination, or error handling. For a read-only listing tool with zero annotation coverage, this is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero waste: the first states the purpose, and the second provides usage guidelines with a clear alternative. It is front-loaded and efficiently structured, earning its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is mostly complete—it explains what the tool does and when to use it. However, it lacks details on behavioral aspects like output format, which slightly reduces completeness for a tool with no structured data to rely on.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description does not discuss parameters, which is appropriate, but it adds value by explaining the tool's purpose and usage context, justifying a score above the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all available routing profiles with their descriptions'), making the purpose specific and unambiguous. It distinguishes this tool from its sibling gateway_set_profile by indicating that one lists profiles while the other switches to them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides when to use this tool ('List all available routing profiles') and when to use an alternative ('Use gateway_set_profile to switch to a profile'), including a named sibling tool and example use cases like 'coding' or 'research'. This gives clear guidance on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_list_serversA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate this is a read-only, non-destructive, idempotent operation with a closed-world scope. The description adds valuable context by specifying what information is returned (status, tool count, circuit-breaker state), which helps the agent understand the output format beyond the safety profile covered by annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('List all connected MCP backend servers') and immediately specifies the returned attributes. There is zero wasted verbiage, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters, no output schema) and rich annotations covering safety and behavior, the description is nearly complete. It adds output details (status, tool count, circuit-breaker state) that compensate for the lack of output schema. A minor gap is the absence of explicit usage guidance versus siblings, but overall it provides sufficient context for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately focuses on the tool's purpose and output details, earning a baseline score of 4 for zero-parameter tools that avoid unnecessary parameter discussion.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List all'), resource ('connected MCP backend servers'), and scope ('with their status, tool count, and circuit-breaker state'), which distinguishes it from siblings like gateway_get_stats or gateway_list_tools that focus on different resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving server status information, but does not explicitly state when to use this tool versus alternatives like gateway_get_stats (which might provide aggregated metrics) or gateway_list_tools (which lists tools rather than servers). It provides clear context but lacks explicit exclusions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_reload_configA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so well. It discloses key behavioral traits: it triggers an immediate action, returns a summary of changes, and notes limitations (server host/port changes are reported but not applied). This covers mutation effects and output behavior adequately.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core action, followed by return details and limitations. Every sentence adds value without redundancy, making it efficient and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a mutation with no annotations and no output schema), the description is mostly complete. It explains what the tool does, what it returns, and key constraints. However, it could briefly mention error cases or prerequisites, but the coverage is strong for the context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so the baseline is 4. The description does not need to add parameter details, and it appropriately focuses on the tool's behavior and output instead.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('trigger an immediate reload of config.yaml from disk') and resource ('gateway'), distinguishing it from siblings like gateway_kill_server or gateway_revive_server. It goes beyond the title by specifying the config file and the no-restart requirement.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context on when to use this tool (to reload config without restarting) and implicitly when not to use it (for server host/port changes, which require a restart). However, it does not explicitly name alternatives like gateway_kill_server for full restarts or compare to other config-related tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_search_toolsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable context beyond annotations: it specifies that results are ranked, include full schemas, save ~95% context tokens, and support multi-word queries and synonym expansion. Annotations cover read-only, non-destructive, and idempotent traits, but the description enriches this with performance and functionality details without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and key benefits in two concise sentences. Every sentence earns its place by adding specific value: the first states the action and efficiency gain, the second details query support. No wasted words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (search functionality with ranking and schema retrieval), rich annotations (read-only, idempotent, etc.), and the presence of an output schema, the description is complete enough. It covers purpose, usage, behavioral traits, and efficiency benefits, leaving output details to the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents the 'limit' and 'query' parameters. The description adds no additional parameter semantics beyond what the schema provides, such as details on query syntax or result ranking, but it does imply the 'query' parameter supports multi-word and synonyms, which slightly enhances understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb ('Search') and resource ('tools'), and distinguishes from siblings like 'gateway_list_tools' by emphasizing keyword-based search with ranking and schema retrieval. It explicitly mentions the efficiency benefit of saving context tokens versus loading all definitions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool: for searching tools by keyword with ranked results and full schemas, and when not to use it (implied by contrasting with 'gateway_list_tools' for listing all tools without search). It also mentions the benefit of saving context tokens, guiding usage based on efficiency needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gateway_list_disabled_capabilitiesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so effectively by disclosing key behavioral traits: it explains the cause of disabling ('high error rate'), the auto-recovery mechanism ('auto-recover after the configured cooldown period'), and the default cooldown time ('default 5 min'), though it could mention output format or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by additional context in a second sentence, and ends with a usage guideline—all sentences are essential with zero waste, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (listing disabled capabilities with recovery details) and no annotations or output schema, the description is mostly complete: it covers purpose, behavior, and usage. However, it lacks details on the output structure (e.g., format of entries) or potential errors, leaving a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Since there are 0 parameters and schema description coverage is 100%, the baseline is 4. The description adds no parameter information, which is appropriate here, but does not detract from the schema's completeness.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List capabilities that have been automatically disabled due to a high error rate') and resource ('capabilities'), distinguishing it from siblings like gateway_list_servers or gateway_list_tools by focusing on disabled capabilities rather than general listings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use this tool ('List capabilities that have been automatically disabled') and provides a clear alternative for manual re-enablement ('Use gateway_revive_server to manually re-enable an entire backend immediately'), guiding the agent on tool selection based on the desired outcome.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.