Server Quality Checklist

Tool Scores

• orcarouter_chatA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (no readonly, not destructive), the description details automatic max_tokens routing for reasoning models, fallback chain behavior, error representation (isError:true), common failure cases, and required API key. This enriches agent understanding of side effects and operational constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single paragraph of 5 sentences, front-loaded with the primary purpose. Every sentence adds essential information (model routing, fallback, error handling, auth) without redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description covers input usage, error handling, authentication requirements, and behavioral nuances. The sibling tools are all list/info tools, so this tool's role is clearly the primary action tool. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for all 6 parameters. The description adds value beyond schema by explaining the automatic max_completion_token routing for reasoning models and confirming the fallback chain behavior, which is not fully detailed in the schema's parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Send a single-turn chat request to OrcaRouter and return the assistant's response text,' specifying verb (send), resource (chat request), and outcome. It distinguishes from siblings (model_card, models_list, providers_list) which are information retrieval tools, not action tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use default model vs. other routers or direct providers, the fallback chain, error handling, and authentication. It lacks explicit 'when not to use' but the sibling tools are clearly different, so usage context is well established.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orcarouter_models_listA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, non-destructive behavior. The description reinforces 'Read-only, no API key required' and explains filter composition behavior, adding transparency beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (4 sentences), front-loaded with purpose and output details, and contains no unnecessary information. Every sentence is informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description details the returned fields (id, name, description, owned_by, context_length, supported_endpoint_types, pricing). It covers filtering behavior and prerequisites, making the tool fully understandable for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers all 3 parameters with descriptions (100% coverage). The description adds value by explaining filter composition, server-side filtering, and provider id format (lowercase), which enhances meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists LLM models from the OrcaRouter catalog, specifies included fields, and distinguishes itself from sibling tools like orcarouter_chat, orcarouter_model_card, and orcarouter_providers_list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains filter composition (all conditions must match), server-side filtering, and suggests using orcarouter_providers_list first. It lacks explicit when-not-to-use guidance but provides sufficient context for appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orcarouter_model_cardA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Adds error behavior ('isError:true with hint on not found') and confirms read-only and no API key, consistent with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Efficient, front-loaded sentences with no waste; each sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given single parameter, no output schema, and informative annotations, description covers all necessary context including error and auth.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers parameter description fully; description adds practical guidance on using exact id from sibling tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it gets detailed info about a single model, listing specific fields. Differentiates from sibling orcarouter_models_list by purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use when model id is known and full details are needed, and directs to orcarouter_models_list for browsing/filtering.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orcarouter_providers_listA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, etc. The description adds 'no API key required' and 'returns the same list until catalog changes,' providing additional context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences with no redundancy. All essential information is front-loaded and each sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters, no output schema, and clear annotations, the description provides all necessary context: purpose, usage flow, and behavioral properties (caching, auth requirements).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and the description correctly notes that it takes no parameters. The schema is fully covered; no further parameter semantics needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that it lists all model providers with specific fields (provider_id, display_name, icon_url, model_count). It also distinguishes itself from sibling tools by noting that it should be called first to discover provider IDs for orcarouter_models_list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to call this first to discover provider IDs and then pass them to orcarouter_models_list. It also mentions that it takes no parameters and returns the same list on every call.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.