Server Quality Checklist

Tool Scores

• chatA
  3.6/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description contradicts the annotation readOnlyHint=true by stating it sends a chat message, which is a write operation. Despite providing additional behavioral details like auto-healing, the contradiction reduces transparency score to 1 per rules.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences long, front-loads the main action, and every sentence adds value without redundancy. It is highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers key aspects: what it does, routing, validation, auto-heal, and return type. However, it lacks details on error handling and the exact structure of the validation report. Given no output schema, slightly more detail would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3. The description does not add new meaning beyond the schema; it only summarizes the tool's behavior. No per-parameter elaboration.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: sending a chat message with automatic output verification. It distinguishes itself from sibling tools (health, providers, stats, validation_history) which serve different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool (for chat with verification) but lacks explicit guidance on when not to use it or alternatives. The context is clear, but no exclusions or comparisons are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validation_historyA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnly, idempotent, non-destructive. Description adds that buffer holds 500 records and older entries are automatically overwritten, providing valuable behavioral context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no waste: first sentence states purpose and ordering, second lists return fields, third explains pagination. Front-loaded with key action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (pagination, no output schema), the description covers return fields, pagination details, and buffer limit. Annotations handle safety. Complete enough for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions already cover limit and offset fully (100% coverage). The description reiterates defaults and max but adds minimal new meaning (e.g., offset for skipping from most recent). Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool queries recent validation results with pagination, returns newest first, and lists specific fields. It distinguishes from sibling tools (chat, health, providers, stats) by focusing on validation history.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly mentions using limit and offset for pagination, default 20, max 100. Does not state when not to use, but sibling tools are unrelated, so guidance is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• providersA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is clear. The description adds no behavioral traits beyond listing contents; it doesn't disclose authentication needs or rate limits, but these are not critical for a read-only listing tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first defines the tool's output, second explains usage. No redundant information. Front-loaded with purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description enumerates the returned items: configuration details, default models, base URLs, and current status. This is sufficient for an agent to understand the tool's return value. With zero parameters and good annotations, the description is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters are defined (0 params), and schema coverage is 100% by absence. The description correctly avoids parameter details. Baseline 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all supported LLM providers with specific details like configuration, default models, base URLs, and status. The verb 'List' and resource 'supported LLM providers' are unambiguous, and it differentiates from siblings like chat, health, stats, and validation_history.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use the tool: to see provider availability, default models, and custom base URLs for proxy/mirror setups. While it doesn't mention when not to use it, the context is clear and sufficient given the tool's simplicity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint true, so the description adds value by explaining the specific statistics returned, including self-healing events, which implies the tool reports on automated recovery actions. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no fluff. The first sentence clearly states the purpose and outputs, the second gives usage guidance. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no parameters, no output schema, and rich annotations, the description sufficiently explains what the tool returns (specific stats) and when to use it. Could mention if a session must be active, but the guidance 'after a working session' implies it works post-session, so adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and schema description coverage is 100% trivially. Per guidelines, baseline is 4 for no parameters. The description appropriately focuses on what the tool returns.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Show' and resource 'Correctover session statistics', listing the included metrics. It clearly distinguishes itself from sibling tools like chat, health, providers, and validation_history by focusing on aggregate session-level statistics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states to use the tool 'after a working session to review reliability metrics and see how many self-healing events occurred', providing clear context for when to invoke it. However, it does not mention when not to use it or compare with alternatives like the health tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• healthA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint, idempotentHint, destructiveHint; the description adds return info (list of active providers, models, session stats) but no additional behavioral traits beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states purpose, second provides usage guidance. No redundant information, front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless tool with comprehensive annotations, the description fully explains what the tool returns and when to use it. No gaps given the simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters defined, so baseline is 4 per guidelines. Description does not need to add parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Check' and the resource 'health and availability of all configured LLM providers', differentiating from siblings like 'providers' which might list all providers without health status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells the agent to 'Call this first to verify your configuration before using the chat tool', providing a clear when-to-use recommendation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.