Server Quality Checklist

Tool Scores

• recommend_deploymentC
  2.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish read-only, idempotent, non-destructive behavior. The description adds useful context beyond that: the operation costs '$0.25 USDC', and the recommendation is deliberately bounded to 'StackBench fit, performance, confidence, evidence, caveats, and economics only when supported', suggesting conservative behavior. There is no contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single compact sentence with no filler, and the fee is mentioned up front. However, it is dense and relies on unexplained domain jargon such as 'bounded StackBench' and 'only when supported', which slightly undercuts its structural clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 42-parameter tool with 0% parameter coverage, this description is far from complete. Annotations and output schema help, but the description still does not tell an agent which parameters are central, what 'recommend what to buy or run' produces, or how this differs from sibling tools. It is a minimal selection-level description, not an invocation-level one.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% across 42 parameters, and the description does not compensate. It neither explains the required model and prompt_tokens inputs nor any of the many mode, policy, threshold, or constraint parameters. The conceptual list in the description does not map to any specific schema property, so an agent gets almost no parameter-level meaning beyond names and enums.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'recommend what to buy or run', and it lists the main inputs the recommendation is based on (fit, performance, confidence, evidence, caveats, economics). It is reasonably distinct from siblings, but the phrase 'from bounded StackBench... only when supported' is jargon-heavy and does not explicitly differentiate it from generate_launch_config or predict_performance.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given about when to use this tool instead of check_deployment_fit, predict_performance, search_evidence, or generate_launch_config. The description implies a synthesizing recommendation role, but it never states conditions, exclusions, or alternatives, so an agent has to infer the decision boundary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_deployment_fitB
  3.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already disclose readOnlyHint, idempotentHint, and destructiveHint, so the description need not repeat those. It adds valuable context beyond annotations by mentioning a $0.01 USDC cost and 'bounded' evidence, which are meaningful behavioral traits. It does not disclose rate limits or authentication requirements, but the cost disclosure is significant.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no fluff, making it highly concise. The cost prefix is somewhat distracting, but the overall structure is compact and front-loads a key operational detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 7 parameters and no schema descriptions, the description is too sparse to be complete. The output schema may cover return values, but missing usage guidance and parameter semantics leave an agent with insufficient information to invoke the tool correctly and confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for undocumented parameters, but it does not. 'Model/configuration' vaguely maps to model, hardware_system, and related fields, yet parameters like concurrency, quantization, engine, and max_evidence are left entirely unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('check model/configuration fit') against 'bounded StackBench capacity evidence,' which distinguishes it from sibling tools like search_evidence or recommend_deployment. However, the '$0.01 USDC on Base mainnet' prefix and the jargon-heavy 'bounded StackBench capacity evidence' slightly obscure the core purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives such as predict_performance or generate_launch_config. The description implies a validation/checking use case but never states conditions or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_launch_configA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already communicate read-only, idempotent, and non-destructive behavior. The description adds value beyond that by disclosing the $0.50 USDC cost, stating that generation is deterministic, and explicitly clarifying that no launch claim is made—important side-effect and cost context for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that front-loads the most decision-relevant facts: cost, determinism, output type, and the critical no-launch disclaimer. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema and safety-related annotations, the description does not explain what an 'exact StackBench recommendation' looks like, how the required parameters should be provided, or what the generated templates contain. For a tool with 19 parameters and zero schema coverage, this is materially incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description bears the full burden of explaining parameters. It only hints at 'vLLM or llama.cpp', which maps to the runtime enum, and leaves the other 18 parameters, including required ones like model, hardware_system, and prompt_tokens, completely unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('generate') and resource ('vLLM or llama.cpp launch templates') and names the required input ('an exact StackBench recommendation'). It clearly distinguishes this from sibling tools like recommend_deployment by indicating it creates launch templates rather than producing a recommendation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'from an exact StackBench recommendation' supplies a clear precondition: this tool is appropriate only when such a precise recommendation exists. It does not explicitly name alternatives for other cases, but the contextual constraint is strong enough for an agent to route correctly in most situations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• predict_performanceA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the tool read-only and idempotent. The description adds meaningful context beyond that by revealing the cost ($0.05 USDC), the bounded-evidence scope, and the output characteristics (confidence, missing metrics, caveats). No contradiction with the annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loads the cost, and conveys the verb, scope, outputs, and caveats without redundancy. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 15 parameters and zero schema descriptions, this one-sentence description is insufficient for correct invocation. An agent cannot determine the meaning of bounded evidence, the semantics of max_evidence, decode_context_depth, or the kv_cache fields, and the description does not clarify how this tool relates to siblings. The output schema reduces the need to explain return values, but parameter selection remains underspecified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the 15 parameters. It only indirectly maps predicted outputs (throughput, memory, power) to the requested_metrics enum, but gives no guidance on model, hardware_system, prompt_tokens, or optional knobs like max_evidence, quantization, and kv_cache_type. The description fails to compensate for missing schema parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('predict') and a clear resource ('configuration throughput, memory, and power'), and it signals the prediction is based on bounded StackBench evidence with confidence and caveats. This distinguishes it from siblings like search_evidence and recommend_deployment, so an agent understands which function is being offered.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'from bounded StackBench evidence' implies the tool is meant for prediction grounded in existing evidence, and the mention of 'confidence, missing metrics, and caveats' suggests when its output is appropriate. However, there is no explicit when-to-use guidance, no exclusions, and no reference to alternatives such as recommend_deployment or check_deployment_fit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_evidenceA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety. The description adds useful context about cost ('$0.00 free') and scope ('bounded public StackBench benchmark and failure evidence'), but it does not disclose additional behavioral traits such as rate limits, query interpretation, or result format; the output schema helps fill the return-format gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The entire description is one efficient sentence with no wasted words. It front-loads the action and scope, then states the usage condition. Every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter search tool with a rich annotation set and an output schema, the description is largely sufficient: it states scope, cost, and when to use it. The main gap is parameter-level instruction, but given the simple schema and available output schema, the overall context is adequate for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it does not explain query semantics or limit behavior. 'Query' is implicitly a search term, but there is no guidance on how to phrase queries or how 'limit' affects results. The two parameters remain mostly self-explanatory only by their names, not by meaningful description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('search'), a specific resource ('bounded public StackBench benchmark and failure evidence'), and the intended use ('evidence-only requests'). It clearly distinguishes this tool from the deployment/prediction/configuration siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'use this for evidence-only requests,' giving the agent a clear condition for when to select this tool. It does not name alternatives or explicitly exclude other cases, but the usage context is clear from the sibling tool names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.