Server Quality Checklist

Tool Scores

• check_numeric_claimsC
  2.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses specific detection algorithms (round-number ratio, MAD-based outlier detection), conditions for compound growth detection, and optional iteration-based escalation/stall detection. It does not describe return format or error behavior, but for a read-only analysis tool this is fairly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with a summary and numbered layers, but it is repetitive (mentions optional context twice) and includes a misleading example that takes up space. The required input format is unnecessary and incorrect. Several sentences fail to earn their place, making it less concise than it appears.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers analysis layers but fails to map parameters accurately to the schema, omits the 'description' field, and provides no return/output information. Given the tool has a nested object parameter and no output schema, the description is incomplete and partially misleading, leaving critical usage details unresolved.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description directly contradicts the input schema: it labels 'context' as a string, but the schema defines it as an object with nested properties. It also completely omits the 'description' parameter, which the schema indicates enables compound growth detection. The example JSON uses 'context' as a string, misleading agents into constructing invalid requests. This actively harms parameter understanding despite high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs multi-signal numeric analysis with specific layers (fabrication detection, outlier detection, arithmetic verification), giving a specific verb+resource. It distinguishes itself from sibling verify_arithmetic by adding fabrication and outlier detection, though it doesn't explicitly name that alternative.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool vs alternatives like verify_arithmetic or validate_confidence. The description focuses on input format and analysis layers, but does not explain selection criteria, prerequisites, or exclusions. Usage is only implied by the tool name and analysis scope.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• detect_concurrency_patternsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full transparency burden. It discloses the types of patterns detected and the input structure, but does not mention output format, error behavior, or limitations. This is a moderate disclosure for a pure analysis tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably structured with a clear lead sentence and a required input format section. The example and bullet list are useful, though slightly long. Every part serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description thoroughly explains the input format and the detection scope, but lacks any description of the return value or output structure. Since no output schema exists, this is a notable gap for a tool whose purpose is to produce analysis results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents all five parameters (100% coverage), but the description adds a concrete JSON example and clarifies how 'shared_resources', 'protections', 'delivery_model', and 'retry_behavior' fit into the detection context. This goes beyond the schema's simple descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Detect'), names the resource ('common concurrency hazard patterns'), and specifies the input ('structured operation description'). It also enumerates specific patterns (check-then-act, read-modify-write, missing idempotency, ordering assumptions), clearly distinguishing it from sibling validation/analysis tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a required input format and optional fields, but does not explicitly state when to choose this tool over alternatives or when not to use it. Usage is implied by the concurrency focus, but no exclusions or alternative tools are named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• detect_driftA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the CUSUM formula (S_i = max(0, S_{i-1} + x_i - omega)), the drift threshold (S_i > 5 * std(sequence)), and the output of monotonic progress (is_improving, is_stalling, is_declining). However, it does not explicitly state the return format or side effects (e.g., read-only), though the analysis-oriented nature is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: it begins with a clear purpose, then provides a required input format example, then the formula, then optional fields. It is somewhat long but every sentence contributes, and it is front-loaded with the key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Without an output schema, the description should explain what the tool returns. It mentions 'reports monotonic progress' and lists the fields, and states when drift is detected, but it does not provide the exact output structure (e.g., a JSON object with drift_detected, is_improving, etc.). The context parameter is also complex, but the description gives only a high-level hint. Given the complexity, this is a notable gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds a concrete example with values, notes the default for drift_sensitivity (0.5), and explains context's purpose (escalation and stall detection). However, the schema already provides detailed descriptions for all parameters, so the description adds marginal value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Detect drift in a numeric sequence using CUSUM (Cumulative Sum) analysis with monotonic progress tracking.' This clearly states the specific verb (detect), resource (numeric sequence), and method (CUSUM), and distinguishes it from sibling tools like validate_confidence and check_numeric_claims, which focus on validation rather than drift detection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a required input format example and explicitly notes optional fields (drift_sensitivity, context) with their defaults and purpose. It clearly implies when to use it (for drift detection in numeric sequences) but does not explicitly mention alternatives or exclusion scenarios. This is clear context without formal exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_reasoning_chainA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and does well: it lists the returned analysis (cycles, orphaned conclusions, grounding_score, enforcement results) and explains that passing context enables escalation and stall detection. It doesn't detail the exact output structure or error handling, but the behavior is well disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description opens with the core purpose and then organizes required input, returns, and optional context. The JSON example is long but essential for a graph-shaped input. No filler sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description lists all return categories and explains the optional context parameter for iterative use. It could specify the shape of 'enforcement results' more concretely, but overall it covers the tool's complexity effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers all parameters extensively (100% coverage), but the description adds a concrete JSON example and clarifies node types and edge relations beyond the schema enums. It also explains the optional context's role, adding meaning beyond parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first line clearly identifies the action ('Map your reasoning to a directed graph and check it for logical errors') and specifies the error types detected (circular reasoning, unsupported conclusions, orphaned claims). This is a precise verb+resource that distinguishes it from sibling tools focused on confidence or arithmetic.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description sets clear context for use: validating reasoning chains by converting them into a directed graph. It does not explicitly name alternatives or exclusions, but the focused purpose and input format make the intended usage clear. A slight deduction for not addressing when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_arithmeticA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Although no annotations are provided, the description discloses key behavioral traits: strict default matching to 2 decimal places and the optional 'tolerance' parameter for relative tolerance. It does not mention return format or error behavior, but the disclosed details are valuable and go beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear example, a bulleted list of claim types and required fields, and concise statements about default strictness and tolerance. Every sentence contributes necessary information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (9 parameters, 5 claim types), the description covers supported operations, required fields per type, and matching behavior. The only notable omission is the return value/output format, but the description is otherwise complete enough for an agent to call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds significant value by specifying which fields are required for each claim_type and providing a concrete example. This clarifies parameter usage beyond the schema's field-level descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Verify that a claimed arithmetic result matches the actual computation.' It lists specific supported claim types (sum, weighted_average, percentage, growth, product), which distinguishes it from sibling tools like check_numeric_claims or validate_confidence that focus on broader validation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on how to structure inputs for each claim type, including a required JSON format and per-type field requirements. It does not explicitly contrast with sibling tools, but it offers clear context on when to use this tool (for verifying arithmetic claims) and how to construct calls.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_plan_validityA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It transparently discloses the validation checks performed, the return fields, and the optional context behavior for escalation/stall detection. It does not explicitly state side effects (e.g., read-only), but for a validation tool the described behavior is sufficiently clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the purpose. The required input example occupies space but earns its place by clarifying the exact structure. Every section (purpose, input format, returns, context) is concise and information-dense, with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (nested objects, optional context), the description covers the essential aspects: input format, semantics, return fields, and context purpose. There is no output schema, so listing return fields is helpful, though it does not detail the types/meaning of completeness_score or critical_path. Overall, it is complete enough for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although schema description coverage is 100%, the description adds substantial value by providing a concrete 'REQUIRED INPUT FORMAT' example with exact JSON structure, clarifying how to represent empty dependencies and optional resources. It also explains that resources are used to detect conflicts, which is not in the schema. This goes well beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'Validate a plan's logical structure: detect circular dependencies, missing prerequisites, and resource conflicts.' This distinguishes it from sibling validation tools like check_numeric_claims or verify_arithmetic, and the concrete list of checks makes the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description establishes clear context for when to use the tool (validating plan logical structure) through its opening sentence and input format instructions. However, it does not explicitly mention alternatives or exclusions, so it falls short of a 5. The guidance is still more than implied because it explains exactly what kinds of plans and structures it validates.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_confidenceA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full behavioral burden. It explicitly discloses the algorithm (dependency-weighted ceiling), the inflation threshold (>0.15), the falsifiability check, and the rule about confidence ≤0.3 when no falsification condition exists. This is highly transparent and goes beyond a simple operation summary.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured and front-loaded with the purpose. The required input format example is somewhat long but earns its place by reducing ambiguity. Every section serves a distinct role (purpose, input format, rule, algorithm, optional context). Slightly verbose due to the JSON example, but efficient overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description should clarify return values. It describes the behavior (computing ceiling, flagging inflation) but does not explicitly state the output format. However, the algorithm description makes the output inferable, and the parameter schema is fully covered. The optional context for iterative enforcement is also explained at a high level, making it complete enough for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds extra semantic value by providing a concrete JSON example showing the exact structure of assumptions and response_text, and by adding a constraint that falsification_condition is mandatory unless confidence is set to ≤0.3. This goes beyond the schema's basic property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb+resource: 'Check whether your claimed confidence is mathematically supported by your assumptions.' It also explains the core function (computes dependency-weighted confidence ceiling, flags inflation, checks falsifiability), which distinguishes it from sibling tools that focus on reasoning chains, numeric claims, or overall quality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: validate a confidence claim against explicit assumptions with falsification conditions. It provides a required input format and a rule for when to lower confidence (if no falsification condition, set ≤0.3). No exclusions or comparisons to alternatives are given, but the guidance is clear enough to know when to invoke this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• evaluate_tradeoffsA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully carries the behavioral disclosure burden. It reveals the EU calculation, ranking behavior, and the INDETERMINATE return (recommended=null) when the top-2 scores differ by <0.05, as well as the purpose of context for escalation/stall detection. This is thorough for a pure computation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core function, then provides an exact input template, validation rules, and edge-case behavior. The JSON example is lengthy but earns its place as a 'copy this structure exactly' requirement. No filler content is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a complex nested schema and no output schema, so the description must compensate. It explains the key return behavior (indeterminate with null recommendation), but does not fully specify the output structure (e.g., what fields are returned for ranked options). The context parameter's effect on escalation/stall detection is mentioned but not detailed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and includes descriptions for all parameters, so the baseline is 3. The description adds a concrete JSON example and clarifies the probability tolerance (+/-0.01) and minimum options, enhancing practical understanding beyond the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Compare options by computing Expected Utility (EU) for each, then rank them,' which clearly states the tool's verb (compare), resource (options), and method (EU, ranking). It distinguishes this tool from sibling validation/checking tools by focusing on tradeoff evaluation rather than confidence or reasoning validation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies the required input structure and validation constraints (probabilities sum to 1.0 ±0.01, minimum 2 options) and explains when to use the optional context parameter for escalation/stall detection. However, it does not explicitly mention alternatives or when-not-to-use scenarios, so it lacks exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score_response_qualityA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full behavioral disclosure. It thoroughly explains the scoring dimensions, the weighting formula, and specifically states that it 'Returns the weakest dimension with targeted improvement advice.' It also discloses the optional use of context for escalation and stall detection, leaving no ambiguity about tool behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose. The required input format is presented in a clear code block, followed by concise bullet-like explanations of dimensions and weighting, and a closing note on optional context. Every section earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema exists, so the description appropriately explains the return value ('Returns the weakest dimension with targeted improvement advice'). It also covers all necessary behavior, including the scoring method and optional iterative context, making the tool fully understandable for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though schema coverage is 100%, the description adds considerable meaning beyond the schema. It provides the exact required input format as a copyable JSON structure, clarifies that 'claims' and 'evidence' are optional, and explains how the 'context' parameter enables escalation and stall detection. This goes far beyond the schema's property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Score a response across four quality dimensions: substance, specificity, hedge avoidance, and structure.' This clearly differentiates it from sibling tools that validate confidence, reasoning chains, or numeric claims by focusing on overall response quality across defined dimensions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by defining what the tool scores and mentions optional context for iterative enforcement, but it does not explicitly state when to use this tool versus alternative validation tools, nor does it provide when-not-to-use guidance. The context paragraph hints at iterative workflows but lacks direct alternatives or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.