Server Quality Checklist

Tool Scores

• compare_modelsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It does not disclose behavioral traits such as whether the operation is read-only, potential side effects, or output format. The minimal description lacks sufficient transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short with no wasted words. It lacks structure (e.g., sections) but remains efficient. A bit more detail could be added without compromising conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 param) and presence of output schema, the description is incomplete. It fails to describe what 'side by side' means in the output, whether it shows differences or full models, or how results are presented.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains that model_ids is 'comma-separated model IDs,' adding format context beyond the schema (which only specifies type string). However, with 0% schema description coverage, more detail on parameter constraints would improve clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Compare multiple models side by side,' specifying the verb 'compare' and the resource 'multiple models.' It distinguishes itself from sibling tools like get_model (single) and list_models (list) by implying a comparative operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives. While the purpose implies it's for comparing multiple models, there is no guidance on exclusions or when-not-to-use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the filtering and sorting parameters, implying a read-only listing operation. However, it does not mention rate limits, pagination, result limits, or any side effects. For a simple list tool, basic behavioral traits are partially covered but not comprehensively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with a clear one-liner purpose followed by parameter details in a readable arg list. No unnecessary words. However, the parameter list could be formatted more clearly (e.g., bullet points) for machine parsing, though it remains human-readable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 optional params, output schema exists), the description is functionally sufficient but lacks context about when to invoke it relative to siblings. It does not mention that it returns a full list or the default behavior (e.g., all modalities). The output schema likely covers return format, but usage context is minimal.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description adds meaning by listing possible values for 'modality' (text, image, audio, embeddings, all) and 'sort_by' (name, created, price, context_length). However, it does not explain what each sort option means (e.g., alphabetical, date, cost, token limit), leaving some ambiguity. The added value compensates for schema gaps but is still minimal.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List models available on OpenRouter.' This directly distinguishes it from siblings like 'compare_models' (comparison), 'get_model' (specific model), and 'search_models' (search). The verb 'List' plus resource 'models' is specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus alternatives (e.g., search_models, compare_models). It only describes the parameters, leaving the agent to infer the use case. Without explicit context, the agent may struggle to choose the appropriate tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_modelA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It states a simple read operation with no side effects, which is accurate but lacks details on potential errors (e.g., if model_id doesn't exist) or caching behavior. The safety profile is implied but not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise – one line plus a parameter note – without wasted words. While it lacks formal structure, it efficiently conveys the essential information for a simple getter tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter and an output schema, the description covers the core purpose and parameter explanation. Minor omissions like error handling or existence checks could be included, but overall it is sufficient for a straightforward operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description adds an example value for model_id ('anthropic/claude-sonnet-4.6') and clarifies it must be a model slug. This significantly improves understanding beyond the schema's bare 'Model Id' title.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves detailed info for one model, using a specific verb ('Get') and resource ('detailed info for one model'). It distinguishes itself from siblings like list_models (multiple models) and compare_models (comparison).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives like list_models or search_models. The description only states what it does, leaving the agent to infer context without any directional cues.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_modelsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden, but it only explains parameter semantics. It does not disclose side effects, authentication requirements, rate limits, or how filters combine. The tool's effect on the system is opaque.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose and then lists parameters in a clear, compact format. Every line provides essential information without redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the 7 parameters and the existence of an output schema, the description covers parameter purposes but omits context on output format, default behavior, and limitations. It is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the description adds crucial meaning for each parameter, e.g., 'Free-text search in model name/id/description'. It compensates well for the schema's lack of descriptions, though individual parameter explanations are brief.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search and filter OpenRouter models,' identifying both the action (search and filter) and the resource. It is distinct from sibling tools like list_models and get_model, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description lists filters but does not provide explicit guidance on when to use this tool versus siblings. No alternatives or exclusion criteria are mentioned, only implied through the list of parameters.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_cacheA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. The description mentions 'force refresh' but does not disclose potential side effects (e.g., impact on ongoing requests, rate limits, or whether it is idempotent), leaving significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the key action. Every word contributes to the meaning, with no unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and an output schema exists, the description is largely complete for its purpose. It explains the action, and the output schema can document return values. However, it lacks any usage context or behavioral notes that could be useful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the input schema provides complete coverage. Per guidelines, baseline is 4. The description adds no additional parameter meaning, but that is acceptable given no parameters exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool refreshes the model cache from OpenRouter. The verb 'refresh' and resource 'model cache' are specific, and it distinguishes from sibling tools that compare, get, list, or search models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when cache is stale but does not provide explicit guidance on when to use or when not to use, nor does it mention alternatives among siblings. The context is clear but lacks exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.