Server Quality Checklist

Tool Scores

• compare_modelsC
  2.6/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description only says 'compare responses' without disclosing behavioral traits such as whether it is read-only, how errors are handled, or the format of the output (e.g., does it return all responses or a summary?). This is a critical gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence, which is concise but lacks necessary detail. It is not verbose, but the under-specification reduces informativeness, balancing to a mediocre score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that compares models, agents need return format, ordering, and error handling information. The description provides none of this, and with no output schema or annotations, the completeness is poor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema coverage is 100% with descriptions for all three parameters (models, message, max_tokens). The tool description adds no additional meaning beyond what the schema already provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it compares responses from multiple models, distinguishing it from sibling tools like chat_with_model (single model) and list_models (model listing). However, it lacks specificity about the comparison mechanism (e.g., side-by-side display).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like chat_with_model. Does not mention scenarios where comparing responses is beneficial or when a different tool would be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_model_infoC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description does not disclose any behavioral traits, side effects, or what 'detailed information' entails.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded, but lacks detail. Could be more informative without being verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 1 parameter, no output schema, and no annotations, description is too sparse. Does not specify what information is returned.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of parameters with descriptions. Description adds no extra meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it gets info about a specific model. Sibling tools like list_models or compare_models have different purposes, so purpose is clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. No context about prerequisites or use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chat_with_modelB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for disclosing behavior. It only states that the tool sends a message, without explaining whether the operation is idempotent, what side effects exist, rate limits, or the nature of the response. For a tool with no annotations, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no wasted words. However, it is overly brief and could benefit from a brief note on usage or behavior. Still, it achieves clarity without verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is no output schema, the description should explain what the tool returns (e.g., the model's response). It also fails to address the complexity of 5 parameters, such as how system_prompt interacts with message. The description is incomplete for a tool of this nature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, meaning each parameter is already described in the schema. The description does not add extra meaning beyond the schema. Baseline score of 3 is appropriate because the schema handles documentation, but the description contributes no additional parameter context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Send a message to a specific OpenRouter model' uses a specific verb (send) and identifies the resource (message to model). It clearly distinguishes from sibling tools like analyze_document, compare_models, get_model_info, and list_models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description lacks context about prerequisites, when to choose chat_with_model over other tools, or any exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It fails to disclose essential behaviors like whether authentication is required, if the list is paginated, what fields are returned, or any rate limits. The minimal description leaves agents with unclear expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no wasted words, but it sacrifices important information. Conciseness is acceptable, but the lack of structure (e.g., no separation of core behavior from usage notes) reduces usefulness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters, no annotations, and no output schema, the description should provide context about the returned data (e.g., model IDs, names) and prerequisites. It fails to do so, resulting in an incomplete tool definition for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the baseline is 4. The description adds minimal meaning ('Get list of available OpenRouter models') beyond the empty schema, confirming the purpose of the tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get list') and the resource ('available OpenRouter models'), which distinguishes it from siblings like 'get_model_info' (detailed info on a specific model) and 'compare_models' (comparison).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for retrieving the model list but offers no guidance on when to use it versus alternatives (e.g., 'get_model_info' for details, 'compare_models' for comparison). No explicit context provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_documentA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description is the sole source of behavioral info. It mentions parallel processing and multiple instances, which is useful but does not cover potential side effects like cost, latency, or result aggregation. It adds value beyond the schema but remains incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the core functionality without extraneous words. Every element serves a purpose, achieving maximum conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 9 parameters and no output schema, the description provides only a high-level overview. It lacks detail on output format, error handling, or prerequisites, leaving gaps that could hinder correct invocation in complex scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with parameter descriptions, setting a baseline of 3. The tool description adds cohesive context by explaining the parallel chunking approach, which gives meaning to parameters like chunk_size, overlap, and parallel_instances, elevating understanding beyond individual schema entries.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (analyze), the resource (large documents), and the method (parallel processing with multiple model instances). This precisely identifies the tool's function and distinguishes it from sibling tools like chat_with_model, compare_models, etc.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not indicate when to use this tool versus alternatives or when not to use it. It lacks any usage context or prerequisites, leaving the agent to infer applicability without guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.