Server Quality Checklist

Tool Scores

• search_nonprofitsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description does not disclose behavioral traits such as rate limits, authentication requirements, data freshness, or behavior when no results are found. The burden is fully on the description, which falls short.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with action and target, no fluff. Efficiently conveys core purpose and optional filters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Moderate complexity with 5 parameters, no output schema. Description covers purpose and filters but omits return format, pagination details, and error handling. Adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description mentions filters but adds little meaning beyond the schema: it names the database (ProPublica Nonprofit Explorer) but does not clarify parameter formats or constraints not already in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches US nonprofits by name/keyword in a specific database (ProPublica Nonprofit Explorer) and lists optional filters. It distinguishes from sibling tools which are compare, get, and list operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs siblings or alternatives. The description does not mention exclusions or appropriate contexts beyond the basic search operation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_nonprofitsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It mentions 'key financial metrics' and 'most recent filing' but does not disclose what happens if filings are missing, whether the tool is read-only, or any rate limits or error conditions. This is insufficient for behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 15 words. It is front-loaded with the main action and provides all necessary information without any extraneous text. Every word contributes meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter and no output schema, the description provides a basic understanding of input and purpose but fails to describe the output format or behavior for edge cases (e.g., missing data). It is minimally adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers the parameter 'eins' fully (100% coverage) with minItems/maxItems and description. The tool description adds little beyond '2-5 nonprofits' which is already in the schema. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Compare'), the resource ('nonprofits'), the scope ('2-5', 'side by side'), and the content ('key financial metrics from their most recent filing'). It effectively distinguishes from sibling tools like 'get_organization' (single org) and 'search_nonprofits' (search).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly tells when to use (comparing multiple nonprofits) but does not provide explicit when-not-to-use guidance or mention alternative tools. It lacks exclusions like needing exactly 2-5 EINs or what to do for more than 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_organization_filingsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses default behavior and the effect of verbose, but lacks details on limitations, pagination, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two front-loaded sentences. Every word adds value, no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description does not fully detail the return format (what curated summaries contain). It adequately covers the core functionality but lacks completeness about return structure and edge cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the description adds minimal new meaning beyond the schema definitions for ein and verbose. Baseline of 3 is appropriate as it does not degrade but does not enhance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists IRS Form 990 filings for a nonprofit, ordered newest first. It distinguishes from siblings like compare_nonprofits and search_nonprofits by specifying the exact resource and action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on using the verbose parameter for raw arrays versus default curated summaries. However, it does not explicitly state when not to use this tool or compare to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_organizationA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the verbose mode for raw payload and mentions the summary of most recent filings. It does not state if the operation is read-only, but for a detail retrieval it's implicitly safe.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. The purpose is stated first, followed by an optional parameter guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with 2 parameters and no output schema, the description is adequate. It explains what is returned (metadata + Form 990 summary) and the verbose option. It could mention error handling or scope of summary but is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for both parameters. The description adds no new meaning beyond the schema, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get full details' and resource 'single nonprofit by EIN', and specifies what is included (organization metadata and summary of recent Form 990 filings). It distinguishes from sibling tools like 'search_nonprofits' which is for searching, not retrieving by EIN.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a specific EIN and need full details, but does not explicitly state when not to use or mention alternatives. However, the context given ('by EIN') makes usage clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.