Server Quality Checklist

Tool Scores

• get_door_effect_summaryA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It discloses the historical nature, exact denominator, data source, attribution, and non-predictive/non-recommendation status. It does not specify the exact return format or table schemas, but provides substantial behavioral context beyond the name.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but front-loaded with the main finding and key statistic. The methodology, caveats, and attribution add length but each sentence provides distinct useful context; it is not padded or repetitive.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter summary tool with no output schema, the description covers the data source, scope, computation details, and limitations. It falls slightly short by not enumerating the exact response fields or table structures, but the provided context is largely sufficient for an agent to understand what will be returned.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema fully covers parameter semantics. The description appropriately adds context about the fixed dataset and methodology instead of parameter details, matching the baseline for parameterless tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's output as a variance decomposition for 'The Door Effect' with specific statistics and lender comparison tables, making the purpose concrete. However, it never explicitly states an action like 'returns' or 'provides', relying on the tool name to convey the verb.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains what the data is and includes caveats about prediction and recommendations, but it gives no guidance on when to use this tool versus siblings such as get_national_fha_stats or get_lender_denial_stats. No alternatives or exclusion criteria are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_lendersA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the computation method (denominator: actions 1,2,3; loan_type 2), data source, attribution, and explicitly states it is historical and not predictive/recommendatory. This is exemplary transparency for a read-only listing tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose and keeps ancillary details (attribution, disclaimers) at the end. It is slightly verbose but every sentence adds contextual value, and the structure (purpose → use case → caveats → attribution) is logical.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description gives a good sense of what is returned (list of lenders sorted by denial rate or volume) and even provides an example of the data range (1.8% to 78.7%). It lacks explicit output field names, but for a simple listing tool the context is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the description adds no new parameter details. However, the two parameters (sort enum, limit with min/default/max) are fully self-documented in the schema itself. The description mentions sorting by denial rate or volume but does not map to enum values; given the simplicity, the schema suffices.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List covered FHA lenders') and specifies sorting options ('by denial rate (highest/lowest) or volume'). It also mentions the data source (CFPB HMDA 2025) and the time span, distinguishing it from sibling statistics tools that aggregate data rather than listing individual lenders.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a use case ('useful to see how widely the same federal program is applied across doors') and clarifies limitations (not a prediction or recommendation). However, it does not explicitly compare to sibling tools or state when not to use it, leaving some ambiguity for agent selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_state_denial_statsA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, but the description carries the full burden and does so well: it explains the historical HMDA 2025 source, the exact denominator used for the statistics, the 'where published' caveat, and explicitly disclaims individual-level prediction or lender recommendation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by concise source/methodology, a useful caveat, and attribution. Every sentence adds relevant context, and there is no irrelevant verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with no output schema, the description adequately covers inputs, meaning, source, denominator, and limitations. It does not explicitly enumerate the return fields beyond the gap, but the phrase 'FHA denial statistics' plus the denominator detail is enough to make the tool usable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There is only one parameter and the input schema already covers it completely with a description, length constraints, and an example. The description merely repeats 'US state (two-letter code)' and therefore adds no real semantics beyond the schema, so the baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly defines the tool as returning FHA denial statistics for a US state, with a specific two-letter code, plus a defining output feature: the small-loan vs large-loan gap. This clearly differentiates it from siblings like national FHA stats and lender-level stats.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The tool is implicitly scoped by the phrase 'for a US state' and the siblings make it obvious when to prefer national or lender-focused tools. However, it does not explicitly state when to choose this tool over an alternative or describe conditions under which the gap is not published.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_lender_denial_statsA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so thoroughly: it discloses the data source (CFPB HMDA 2025), the exact denominator definition (actions 1,2,3; loan_type 2), the historical nature of the observation, top-100 coverage limits, and explicit caveats that it is not a prediction or recommendation. This is exemplary disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized, with the core purpose and metrics in the first sentence, followed by coverage, methodology, and disclaimers. Every sentence contributes meaningful information, and nothing is redundant or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema and no annotations, the description is self-sufficient: it explains what metrics will be returned, the data source and calculation basis, coverage constraints, and important caveats. The tool is simple with one parameter, and the description fully compensates for the missing structured metadata.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single 'lender' parameter, already stating name, slug, or 20-char LEI. The description repeats these identifiers but adds the context that only top-100 lenders are covered, which mildly supplements parameter understanding without providing new syntactical details beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the tool returns FHA denial statistics for one lender, identified by name, slug, or LEI, and enumerates the exact metrics (actual denial rate, peer-median comparison, mix-adjusted expected rate). It also distinguishes itself from sibling tools by emphasizing 'one lender' and limiting coverage to top-100 FHA lenders.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: for per-lender FHA denial stats, with coverage limited to top-100 lenders. It does not explicitly name alternatives like get_national_fha_stats or get_state_denial_stats, nor state when not to use this tool, but the per-lender focus and coverage limit imply appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_national_fha_statsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well: it explains the data source, calculation basis, denominator, and explicitly states this is not a prediction or recommendation. It omits operational details like rate limits or output shape, but for a read-only stats tool this is strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences and front-loaded with the core result. Every sentence adds value: what it returns, how it was computed, and important caveats/attribution. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters, no output schema, and no annotations, the description is sufficiently complete for an agent to understand the tool's scope, source, caveats, and attribution. The sibling names provide additional context that reinforces the national-vs-lender/state distinction.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the baseline for parameter semantics is 4. The description appropriately adds no parameter-specific detail because none are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns national FHA denial statistics, specifically denial rate and volumes, from the 2025 federal record. This is a specific verb+resource and is distinguished from sibling tools by the 'National' scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The national scope is explicitly clear, which lets an agent infer when to use this tool versus state-level or lender-level siblings. However, it does not explicitly name alternatives or provide when-not-to-use guidance, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.