Server Quality Checklist

Tool Scores

• get_methodology_and_citationA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The tool has strong annotations (readOnlyHint=true, idempotentHint=true, destructiveHint=false) already declaring a safe read-only operation, so the bar is lower. The description adds value by enumerating the kinds of information returned but doesn't disclose anything about format, structure, or whether any fields may be absent. Given robust annotations, this is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, efficient sentence that enumerates all delivered content with no filler or redundancy. Every clause earns its place, listing concrete deliverable items. This is exemplary conciseness for a parameterless retrieval tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter, no-output-schema tool with strong annotations, the description is quite complete—it lists ten distinct content elements. The main gap is absence of guidance about output format/structure, but given zero inputs and strong read-only/idempotent hints, the description covers the essentials well.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4 per scoring rules. The description appropriately describes what data the tool returns rather than wasting space on nonexistent parameters. There are no parameters needing schema or description documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description lists specific content returned (provenance, normalization method, limitations, license, citation, canonical tracker, publisher disclosure), making the purpose clear and distinct from its parameterless siblings. It uses a specific verb+resource ('Return... methodology and citation'). It could be improved by clarifying how this differs from get_program_summary, as both appear to be overview/info tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a retrieval use-case but provides no explicit guidance on when to choose this tool over get_program_summary or whether it complements other lookup tools. There are no exclusions or alternative suggestions. The siblings list suggests some overlap with get_program_summary that is not addressed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_county_summaryA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is fully covered. The description adds that it returns aggregate counts plus the 'five largest' records—useful behavioral context about result scope. However, it doesn't disclose response size, ordering, or what happens if a county has unmatched names.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, dense sentence that enumerates exactly what's returned with zero filler. Efficiently front-loaded with the verb and resource. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a read-only, single-parameter aggregate tool with strong annotations. The description covers the return contents and the county scoping. Minor gap: no mention of behavior for unknown/misspelled counties or outcome when a county has fewer than five providers, but given full annotations and 100% schema coverage, this is near-complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (the county param is fully documented, including the 'with or without County' syntax nuance). The description implies the county filters results but adds no format/validation details beyond the schema. Baseline 3 applies given full schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb+resource+scope: 'Return participating provider-record count, licensed capacity, median record capacity, and the five largest provider records for one California ALW county.' This is specific and distinguishes itself from siblings like get_program_summary (program-scoped) and list_counties (county listing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool—when wanting county-level ALW provider aggregates—but doesn't explicitly exclude alternatives or name when-not scenarios. The 'one California ALW county' phrasing gives clear scoping context. Slight room to name alternatives like get_program_summary as the cross-county alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_facilitiesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so safety profile is well-covered. The description adds useful behavioral context beyond annotations: the 'at most 50 records' return cap and the explicit caveat that results 'do not imply current vacancy.' These are meaningful operational disclosures that help the agent interpret results correctly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, each earning its place. First sentence states the search scope and filter dimensions, second adds the critical return cap and the vacancy caveat. No filler, no redundancy with the schema field names it mentions (name/address/county/city/ZIP map directly to the parameters). Front-loaded with the core verb+resource.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only search tool with 80% schema coverage, 5 optional params, and no output schema, this description is largely adequate. It names all filter dimensions, specifies the 50-record ceiling, and warns about vacancy meaning. The subtle gap: it doesn't note that all params are optional and what happens with no filters (would it return broadly?), and semantics like 'does not imply current vacancy' hint at a data freshness limitation that could be stated more directly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 80% (4 of 5 params documented; 'limit' has no description but has default/min/max). The description enumerates the search dimensions (name/address text, county, city, ZIP) which mirrors the schema. It adds modest value by clarifying 'query' accepts provider number as well, though mostly it restates what the schema already covers. With 80% coverage, baseline is 3; the provider-number mention nudges it slightly higher.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Uses specific verb 'Search' with clear resource 'public DHCS participating-provider records' and concrete filter dimensions (name/address text, county, city, ZIP). Effectively distinguishes from siblings: siblings return program/county summaries and methodology, while this tool searches facility records. The origin (DHCS) and public-scope qualifier add precision.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states the search scope (participating-provider records) and adds the important caveat about results not implying current vacancy, which guides interpretation. However, it does not explicitly tell when to use this tool vs. get_program_summary or get_county_summary, and there's no mention that all parameters are optional or how they combine. Sibling differentiation is implied but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_countiesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint=false, idempotentHint, and non-destructive. The description adds meaningful context beyond annotations: the explicit guarantee that it 'does not fabricate county waitlist values' — a valuable behavioral promise signaling data fidelity. With strong annotations covering the safety profile, this is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that pack useful information: scope, count, data fields, and a fidelity guarantee. Front-loaded with the purpose. Slightly redundant that quantity (15) is conveyed, but it's a meaningful data point, not fluff. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 0-parameter, list-type tool with strong annotations covering safety and openness, the description is complete. It explains what data is returned (counts, capacity) and adds the anti-fabrication guarantee. No output schema exists, so some detail on return structure would be nice but isn't essential for a simple list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Tool has 0 parameters and schema coverage is 100% (nothing to document), so the baseline of 4 applies. The description adds clarifying info about what the output includes (provider-record counts, licensed PEU capacity), which helps an agent know what data to expect without needing parameter docs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Specific verb+resource: 'List' all counties with participating ALW facilities. States the exact count (15), what data is included (provider-record counts, licensed PEU capacity), and explicitly notes a scope fact. Clearly distinguishes from siblings like get_county_summary which targets individual counties.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes it evident this is a top-level enumeration tool (list all counties) versus siblings like get_county_summary (single county) or get_program_summary (program-level). While it doesn't explicitly name alternatives or exclusions, the listing semantics and sibling differentiation are clear enough for an agent to choose correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_program_summaryA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, which fully covers the safety profile for a data-retrieval tool. The description adds meaningful context beyond annotations by noting it returns sources, dates, license, and interpretation cautions—ethical transparency about data provenance. It doesn't describe pagination or return format, but with no output schema and zero params, the description carries most of the burden and does so reasonably.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single carefully constructed sentence that front-loads the core purpose (statewide summary of the ALW program) and then enumerates all returned data elements. Zero wasted words; every term in the sentence adds information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has zero parameters, no output schema, and no nested objects—making it simple. The description enumerates all nine data categories returned, and notes important caveats around sources, dates, license, and interpretation cautions. The main gap is not mentioning any return format or whether the summary is a single object or list, but for a zero-param summary tool this is a minor omission. Could also clarify nothing about methodology, though get_methodology_and_citation exists as a sibling for that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are 0 parameters, and schema description coverage is 100% (trivially, since there are no params to document). With no parameters, the baseline per the rubric is 4, and the description doesn't need to add parameter info. The description's field enumeration effectively substitutes for parameter documentation since the tool takes no inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies the exact resource (California ALW program) and enumerates the exact data dimensions returned: statewide enrollment, waitlist, provider-record count, county coverage, licensed PEU capacity, sources, dates, license, and interpretation cautions. This is a very specific verb+resource with a clear list of what the summary contains, distinguishing it from get_county_summary which presumably covers a single county.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is the statewide-level summary tool, contrasting implicitly with get_county_summary (county-level) in the sibling list. However, it doesn't explicitly state when to use this vs get_county_summary or list_counties. The field list mentioning 'county coverage' and 'statewide' signals scope, but there's no explicit when-to-use guidance or exclusion note.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.