Server Quality Checklist

Tool Scores

• get_questionA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Explains tiered hint reveal behavior, but lacks details on error handling or return structure; no annotations provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Very concise with bullet-point parameter descriptions, front-loaded purpose, and no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With output schema present, return values need not be detailed, but missing mention of invalid inputs or error cases; adequate for simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds meaning beyond schema by providing example for question_id and defining hint_level values 0-3, compensating for 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it fetches one question with tiered hint reveal, distinguishing from siblings like search_questions for searching multiple questions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage for fetching a single question with hints, but does not explicitly state when to use vs alternatives like search_questions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_courseA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description discloses the key behavioral trait that pricing is redirected to a live source. It also lists included information (structure, target band, free-tier info), though it lacks details on error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences with the core action front-loaded, no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple get-by-id tool with an output schema, the description covers the purpose, included data, and a notable processing behavior. No major gaps given the simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds an example of the id format ('amc-foundation') but does not fully describe parameter constraints or format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'Get one course by id' with an example, distinguishing it from siblings like search_courses (which likely returns multiple) and get_pricing_info.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It implies that for pricing data one should use a different tool (pricing redirect to live source) but does not explicitly list alternatives or when to use this vs search_courses.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_questionsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It states that the tool returns only question stems and mentions the sample set. However, it does not disclose potential destructive behavior (likely none), authentication needs, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise and well-structured: a one-line summary, then arg specifications, then return info and cross-reference. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (2 parameters) and the presence of an output schema, the description is sufficiently complete. It covers purpose, parameters, return values, and the relationship to a sibling tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description adds essential meaning. It lists valid values for topic and difficulty and indicates that topic performs a substring filter. This is very helpful, though it could be more precise about matching behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search practice questions' with a specific verb and resource. It distinguishes itself from the sibling 'get_question' by noting that this tool only returns question stems, and hints/solutions require the other tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says to use 'get_question' for hints/solutions, providing a clear alternative. It also lists valid argument values, guiding when to filter by topic or difficulty.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_pricing_infoA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but the description discloses that prices are not stored and users must be directed to the live page. This is an important behavioral trait for a read-only tool, though it doesn't elaborate on other aspects like rate limits or authentication.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two sentences) and front-loads the purpose. Every sentence provides essential information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of parameters and presence of an output schema, the description is adequate. It explains the core functionality and design choice. It could mention the output format, but the output schema likely covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters, the schema coverage is 100%. The description adds meaning by explaining the purpose of the tool, which is necessary since the input schema is empty. It clarifies what the tool does beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'the live pricing source' and explains the design rationale. It is specific to pricing queries, distinguishing it from siblings like get_course or search_questions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'How to answer pricing questions' and instructs to 'always send users to the live page for money amounts,' providing clear context for when to use this tool. While it doesn't mention alternatives, siblings are unrelated to pricing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_coursesA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It states it returns matching courses but doesn't disclose read-only status, pagination, rate limits, or side effects. For a search tool, the lack of behavioral detail is a minor gap; transparency is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is concise with clear sections (Args, Returns). It uses bullet points for parameters and ends with a useful note about pricing. No superfluous text; every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema (context: has output schema = true), the description doesn't need to detail return values. It covers all essential aspects: purpose, parameters with semantics, and a key limitation (no prices). The tool is well-documented for an AI agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage, so description must compensate. It does so excellently: subject lists allowed values ('maths', 'science', 'computer-science'), year_level explains meaning and range (5-12), course_type gives examples and states substring filter. This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search the course catalog' and distinguishes from siblings like get_course (specific course) and get_pricing_info (prices). It provides specific parameters and their values, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description gives explicit guidance: 'Returns matching courses (no prices — see get_pricing_info)' redirects users needing pricing. It specifies parameter types and allowed values (subject enum, year_level range, course_type substring filter). However, it doesn't explicitly state when to use this tool over get_course or search_questions, though implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.