Server Quality Checklist

Tool Scores

• get_engine_schemaA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully bears the burden. It clearly states that the tool returns a lean, compact overview with limited param detail, and that it is token-cheap. No side effects are mentioned, but for a read-only discovery tool, this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, using two sentences to convey purpose, usage flow, and trade-offs. It is front-loaded with the main action and immediately provides context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool (one parameter, no output schema), the description covers the core behavior and workflow well. However, the lack of parameter explanation and the fact that the engine parameter's source is not mentioned reduces completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must explain the 'engine' parameter, but it does not. It only mentions 'one engine' without specifying valid values, format, or how to obtain them. This is a critical omission for an agent selecting the tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it provides a compact overview of one engine's actions, descriptions, required params, and return values. It distinguishes itself from sibling tools like get_action_schema and call_engine by explaining its role in the workflow (after search_engines, before get_action_schema).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises when to use this tool (after search_engines to pick the right action) and when to use alternatives (get_action_schema for full params before call_engine). It also explains the rationale for using this tool (keeping token-cheap).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_action_schemaA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must fully convey behavioral traits. It describes the output contents and a validation behavior (rejection of invalid enums), but does not explicitly state that the operation is read-only, nor does it mention rate limits or authentication needs. The description implies safety through its preparatory nature but lacks explicit behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, with the first sentence listing the tool's outputs in a front-loaded manner and the second providing immediate usage context. Every word adds value, and there is no redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 simple parameters, no output schema, and no annotations, the description covers the return value comprehensively (parameters details, returns, pricing, example_params). It also addresses the usage sequence. However, it could mention potential errors or prerequisites beyond invalid enums, which keeps it from being a 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage for its two parameters (engine, action). The description does not add any specific meaning beyond the schema's property names; it only says 'for ONE engine action' without explaining what values engine or action can take. This is insufficient to compensate for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool retrieves full details for one engine action, listing specific elements like parameters, returns, pricing, and example_params. It distinguishes itself from siblings by explicitly positioning it as a preparatory step before call_engine, and from get_engine_schema or get_catalog which are broader in scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises 'Call this right before call_engine so you send valid params', providing clear when-to-use guidance. It also warns about invalid enum values being rejected, which helps the agent avoid errors. No alternative tools are mentioned but the context strongly differentiates this from siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• call_engineA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the HTTP method (POST), endpoint pattern, return format (uniform envelope), and that failed calls cost no credits. It does not mention idempotency or error codes, but overall is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences, each adding distinct value: purpose/method, prerequisite, authentication, return format and cost. Front-loaded and free of redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a generic API caller with 3 parameters and no output schema, the description covers purpose, method, endpoint, return format, prerequisite, auth, and cost. It is quite complete, though a brief example response would improve it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It explains that 'engine' and 'action' are path components and 'params' is the body, and instructs to get param names from get_engine_schema. It adds context but does not list individual parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool calls a ReefAPI engine action using HTTP POST. It specifies the verb 'Call' and the resource 'engine action', and the sibling tools (get_engine_schema, get_action_schema, etc.) are clearly for schema retrieval, making this tool distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises 'Get param names from get_engine_schema first' as a prerequisite, and explains authentication requirements. While it doesn't explicitly state when not to use it, the context is clear and it provides actionable guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_catalogA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the output is '≈ a few thousand tokens' and instructs the LLM to scan and pick the best engine. This informs the agent about the size and nature of the response. Could mention if there are rate limits or caching but covers key behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is direct and front-loaded with the main purpose. It uses emphasis (ALL CAPS) for important instructions. While somewhat lengthy, every sentence adds value including the usage guidelines and workflow. Minor improvement could be tightening.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains that the output contains all engines with titles grouped by category and its token size. It also provides the steps after using this tool. Does not mention update frequency or caching, but for a catalog retrieval tool, this is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has zero parameters and 100% coverage, so baseline is 4. Description correctly includes no parameter discussion since none exist. No additional information needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs and resources: 'returns the FULL ReefAPI catalog — EVERY engine with its one-line title, grouped by category.' It clearly distinguishes from sibling tools, especially search_engines, by stating its use case as a fallback when search fails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: 'Use this whenever search_engines didn't surface the right engine (or to be sure you didn't miss a better one).' Also includes a clear workflow sequence after picking an engine: get_engine_schema -> get_action_schema -> call_engine.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_enginesA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description fully discloses behavioral traits: it performs a keyword-based search with stem-matching, returns a ranked list of engines with match scores, and is described as a fast pre-filter. It mentions that an empty query returns all engines and that the tool does not modify any state. No contradictions with annotations (none present).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the purpose, but is somewhat lengthy. However, every sentence earns its place by providing necessary details about usage, matching, and fallback. Minor verbosity prevents a perfect score, but it remains efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity, the description is complete. It explains the input, return values (name, title, category, actions, match score), and how to proceed after calling it (use get_engine_schema). It also addresses edge cases (empty query, non-English queries) and integrates well with sibling tools. No output schema is needed as the return structure is described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines a single optional 'query' parameter with no description, but the tool description provides extensive detail: it accepts English keywords or natural-language use cases, advises translation if needed, and explains the matching logic. This significantly adds meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to find the right ReefAPI engine for a task by matching a query. It uses a specific verb ('find') and resource ('engine'), and provides examples to illustrate typical use cases. It also distinguishes itself from siblings by indicating it should be called first, and if the right engine is not found, to use get_catalog instead.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly guides when to use the tool ('Call this FIRST') and when not to (if the engine isn't in results, use get_catalog). It also advises translating non-English queries into English keywords before using the tool, providing clear context for appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.