Server Quality Checklist

Tool Scores

• compare_providersA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that missing providers indicate suppressed data (under 10 respondents) and that output includes positivity scores, benchmarks, and UK averages. No annotations exist, so description carries full burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Front-loaded with main purpose, followed by output explanation, parameter details, and a specific behavior note. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers output structure and suppression behavior, but relies on referencing nss_scores for some parameters, which may not be fully self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Describes key parameters (ukprns as provider ids from search_providers, subject as CAH filter, question as theme filter) but defers subject_level, population, etc. to 'As in nss_scores', assuming familiarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool compares NSS results across providers, distinguishing it from search_providers and nss_scores by focusing on side-by-side comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explains that the comparison is fairer than raw ranking and provides parameter guidance (e.g., default question is 'Theme'), but does not explicitly state when to use alternatives like nss_scores.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• b3_outcomesA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully bears the burden of transparency. It explains data source (OfS), what each indicator represents, benchmark and threshold inclusion, pooled years, latest data retrieval, handling of NULL values with suppression reasons, and exclusions (no application volumes, entry grades, or satisfaction scores).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a summary, bullet-pointed indicator details, and clear argument list. It is front-loaded with the core purpose and each sentence adds distinct value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, output schema exists), the description covers all necessary aspects: return structure, parameter usage, suppression handling, and data limitations. It is comprehensive enough for an AI agent to effectively use the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description thoroughly explains all 7 parameters, including default values, allowed values, examples, and relationships (e.g., split_type and split_value). It also provides context for ukprn (from search_providers) and subject (CAH name or code).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns OfS B3 student outcomes for one provider, listing the three indicators and explaining their meaning. It implicitly distinguishes from siblings like compare_providers and nss_scores by specifying that it is for a single provider and noting that satisfaction scores are covered by NSS tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use the tool (e.g., for a specific provider) and details on filtering, but does not explicitly state when not to use it or compare directly to sibling tools. However, it does mention that satisfaction scores are not included, directing users to NSS tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• leo_earningsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavioral traits: earnings are pre-tax, from matched tax data, exclude self-employed/unemployed graduates, reflect location weighting, and are not causal. It also states what the tool does not provide (application volumes, satisfaction, dropout). This is comprehensive and honest.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with paragraphs for purpose, args, and limitations. It is front-loaded with essential info. While slightly lengthy, every sentence adds value; minor redundancy in repeating 'omit' for parameters could be tightened, but overall efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, detailed output) and presence of an output schema, the description covers return values, limitations, and usage context thoroughly. It explains what the data represents and its caveats, making it complete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description bears full burden. It explains each parameter: ukprn (from search_providers), subject (CAH2 partial match, optional), years_after (1,3,5), tax_year (format example), characteristic_type (default and split options), characteristic_value (value examples). This adds significant meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool returns graduate earnings data for one provider, specifying lower-quartile, median, and upper-quartile annualised earnings at 1, 3, and 5 years by CAH2 subject, with UK-wide context. This is distinct from sibling tools like b3_outcomes (broader outcomes) or nss_scores (satisfaction), so purpose is specific and well-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly guides the agent to always quote median with quartiles and mention graduate counts. Discusses honest limits (pre-tax, excludes self-employed, not causal) and clarifies that other tools cover application volumes and dropout. References search_providers for ukprn, providing clear when-to-use and when-not-to-use context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• nss_scoresA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description fully discloses the survey data nature, the return structure (positivity scores with comparators), and limitations (cannot infer other metrics). This provides comprehensive behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Efficiently structured with a summary sentence, a detailed Args block, and a concluding note on limitations. Every sentence adds value without waste, and the purpose is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters and an existing output schema, the description covers all parameter semantics, usage guidance, and data limitations. It is fully complete for an agent to understand and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description explains each parameter in detail with defaults, examples, and the effect of combinations (e.g., subject filter with subject_level granularity). This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns National Student Survey results for one provider, including positivity scores per question/theme with benchmarks and UK sector averages. This distinguishes it from sibling tools like b3_outcomes, compare_providers, leo_earnings, and search_providers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises that a score without its benchmark is misleading and to always quote both. Also states what the tool cannot provide (application volumes, offer rates, etc.), guiding when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_providersA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but description discloses scope (~450 providers), matching behavior (universities rank above colleges), and limitations (no NSS results for small/new providers).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise 4-sentence description, front-loaded with purpose, then usage guidance, then details and limitations. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter search tool with no annotations, description covers purpose, usage, return value (UKPRN), scope, and limitations. Output schema exists, so return details are covered. Complete for effective tool use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Single parameter 'query' has no schema description, but description explains it accepts name, nickname, or fragment and gives concrete examples, fully compensating for schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Find UK higher-education providers by name, nickname or fragment' with specific verb and resource. Distinguishes from siblings by mentioning UKPRN as key identifier used by other tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises 'Use this first when the user names a university loosely' and provides examples. Also notes limitation: cannot find providers without published NSS results.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.